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ABSTRACT 


In  response  to  Congressional  direction,  a  quantitative,  analytical  review  (a  "meta¬ 
analysis")  was  completed  of  interactive  videodisc  instruction  applied  in  Defense  training 
and  in  the  related  settings  of  industrial  training  and  higher  education.  Over  all  instructional 
settings  and  applications,  interactive  videodisc  instruction  was  found  to  improve 
achievement  by  about  0.50  standard  deviations  over  less  interactive,  more  conventional 
approaches  to  instruction.  This  improvement  is  roughly  equivalent  to  increasing  the 
achievement  of  students  at  the  50th  percentile  to  that  of  students  currently  at  the  69th 
percentile.  An  improvement  of  0.38  standard  deviations  was  observed  across  24  studies 
in  military  training  (roughly  an  increase  from  50th  to  65th  percentile  achievement).  An 
improvement  of  0.69  was  observed  across  14  studies  in  higher  education  (roughly  an 
increase  from  50th  to  75th  percentile  achievement).  Interactive  videodisc  instruction  was 
more  effective  the  more  the  interactive  features  of  the  medium  were  used.  It  was  equally 
effective  for  knowledge  and  performance  outcomes.  It  was  less  costly  than  more 
conventional  instruction.  Overall,  interactive  videodisc  instruction  demonstrated  sufficient 
utility  in  terms  of  effectiveness,  cost,  and  acceptance  to  recommend  that  it  now  be  routinely 
considered  and  used  in  Defense  training  and  education. 
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A.  PURPOSE 

The  1989  Department  of  Defense  Appropriations  Bill  directed  the  Department  of 
Defense  (DoD)  to  conduct  a  study  on  the  use  of  interactive  videodisc  technology  in  training 
and  education  as  it  pertains  to  effectiveness,  cost-effectiveness,  time  on  task,  retention,  and 
overall  applicability  to  current  and  future  DoD  t  'mint?  and  education  requirements. 
Accordingly,  a  quantitative,  analytic  review-a  "me  s"~of  what  has  been  learned 

about  the  effectiveness  and  cost  of  interactive  vide^O'  auction  was  completed  and  is 
reported  here. 

B.  METHOD 

Empirical  evaluation  studies  comparing  interactive  videodisc  instruction  with 
conventional  instruction  were  collected  from  three  instructional  settings:  military  training, 
industrial  training,  and  higher  education.  Results  from  these  studies  were  then  calculated 
using  effect  size,  which  is  widely  used  in  meta-analytic  reviews  and  which  provided  a 
common,  quantitative  measure  of  instructional  effectiveness  that  could  be  applied  across  all 
studies  considered.  Effect  size  is  a  standardized  index,  measured  in  standard  deviations,  of 
the  difference-in  this  case  the  difference  in  instructional  achievement-between  two 
treatment  groups.  It  is  discussed  further  in  the  body  of  the  paper.  In  this  review,  the  larger 
the  effect  size,  the  more  effective  interactive  videodisc  instruction  was  found  to  be  than 
conventional  instruction. 

C.  FINDINGS 

The  findings  of  this  review  principally  concern  the  instructional  capabilities,  or 
functionalities,  made  available  by  interactive  videodisc  systems,  not  the  hardware  itself. 

I  .  Effectiveness 

Interactive  videodisc  instruction  was  used  successfully  to  teach.  Four 
comparisons  of  interactive  videodisc  instruction  with  a  "placebo"  treatment  in  which  no 
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relevant  instruction  was  given,  resulted  in  an  average  increase  in  achievement  of  1.39 
standard  deviations,  or,  roughly,  an  increase  in  the  achievement  for  50th  percentile  students 
to  about  the  92nd  percentile  of  achievement 

Overall,  interactive  videodisc  instruction  was  more  effective  than 
conventional  instruction.  Forty-seven  comparisons  of  interactive  videodisc  instruction 
with  conventional  approaches  were  identified  for  this  review.  Over  all  instructional  settings 
(military  training,  industrial  training,  higher  education),  instructional  approaches 
(simulation  only,  tutorial  only,  simulation  and  tutorial  combined),  and  outcomes 
(knowledge,  performance,  retention),  interactive  videodisc  instruction  increased 
achievement  an  average  of  0.50  standard  deviations  over  conventional  instruction  (an 
increase  for  50th  percentile  students  to  about  the  69th  percentile  of  achievement). 

Interactive  videodisc  instruction  was  more  effective  than 

conventional  instruction  in  military  training.  Twenty-four  comparisons  of 
interactive  videodisc  instruction  with  conventional  approaches  to  military  maintenance, 
operator,  and  command  training  were  identified  for  this  review.  Over  all  instructional 
approaches  and  outcomes,  they  showed  an  average  increase  in  achievement  of  0.38 
standard  deviations  (an  increase  for  50th  percentile  students  to  about  the  65th  percentile  of 
achievement). 

Interactive  videodisc  instruction  was  more  effective  than 

conventional  instruction  in  higher  education.  Fourteen  comparisons  of  interactive 
videodisc  instruction  with  conventional  approaches  to  instruction  in  higher  education 
settings  (colleges  and  universities)  were  identified  for  this  review.  Over  all  instructional 
approaches  and  outcomes,  they  showed  an  average  increase  in  achievement  calculated 
using  pooled  standard  deviations  of  0.69  standard  deviations  (an  increase  for  50th 
percentile  students  to  about  the  75th  percentile  of  achievement).  The  lower  average  effect 
sizes  found  for  military  training  than  for  higher  education  may  be  due  to  a  focus  in  military 
training  on  reaching  threshold  level(s)  of  achievement  with  minimized  costs  and  time  so 
that  students  who  reach  achievement  criteria  are  sent  on  to  duty  assignments  rather  than 
held  in  the  instructional  setting  as  they  are  in  higher  education. 

Interactive  videodisc  instruction  was  equally  effective  for  both 
knowledge  and  performance  outcomes.  The  average  effect  sizes  calculated  using 
pooled  standard  deviations  for  27  knowledge  outcomes  (facts,  concepts,  and  other 
information  students  acquired)  and  for  20  performance  outcomes  (procedures,  skills,  and 
other  capabilities  students  could  demonstrate)  both  averaged  around  0.35  standard 
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deviations  (suggesting  an  improvement  of  50th  percentile  students  to  about  the  64th 
percentile). 

The  more  the  interactive  features  of  interactive  videodisc  technology 
were  used,  the  more  effective  the  resulting  instruction.  One  study  examined 
this  issue  directly  and  found  greater  achievement  with  greater  levels  of  interactivity.  The 
issue  can  also  be  examined  by  comparing  the  effect  sizes  for  Level  II  videodisc  instruction 
(-.70,  overall)  with  those  for  Level  III  videodisc  instruction  (.50,  overall). 

Directed,  tutorial  approaches  were  more  effective  than  stand-alone 
simulations  in  interactive  videodisc  instruction.  Effect  sizes  averaged  0.70  for 
tutorial  approaches,  0.40  for  combined  tutorial  and  simulation  approaches,  and  0.15  for 
simulations  by  themselves. 

Within-group  variability  was  smaller  in  interactive  videodisc 
instruction  than  in  conventional  instrrction.  In  nearly  every  comparison  of 
interactive  videodisc  instruction  with  conventional  instruction,  achievement  in  the 
interactive  videodisc  instruction  group  was  less  variable  than  in  the  conventional  instruction 
group,  suggesting  there  was  more  equitable  distribution  of  achievement  using  interactive 
videodisc  instruction. 

Interactive  videodisc  instruction  v  z  more  effective  than  computer- 
based  instruction  without  videodisc  interaction.  Both  interactive  videodisc 
instruction  and  computer-based  instruction  have  been  found  to  be  more  effective  than 
conventional  instruction.  However,  the  average  effect  size  of  around  0.69  observed  for 
interactive  videodisc  instruction  used  in  colleges  is  considerably  higher  than  both  the 
average  effect  size  of  0.26  found  in  a  review  of  computer  based  instruction  used  in  colleges 
and  the  average  effect  size  of  0.42  found  in  a  review  of  computer-based  instruction  used  in 
adult  education. 

There  was  little  in  the  reviewed  studies  to  indicate  how  interactive 
videodisc  instruction  achieves  its  success.  The  studies  examined  in  this  review 
did  little  to  indicate  which  features  of  interactive  videodisc  instruction  contribute  to  the 
observed  increases  in  student  achievement.  Additionally,  there  are  many  outcomes-such 
as  speed  of  response,  accuracy  of  response,  attitude  toward  the  subject  matter,  insight, 
transfer,  and  .ctention-that  may  alone  or  in  some  combination  become  the  objective(s)  of 
instruction.  How  different  designs  for  interactive  videodisc  instruction  contribute  to 


accomplishing  these  different  outcomes  was  rarely  addressed  by  the  reviewed  studies  and 
remains  a  proper  topic  for  future  research. 

2.  Cost 

Interactive  videodisc  instruction  was  less  costly  than  conventional 
instruction.  All  13  cost  ratios  (the  ratio  of  costs  for  interactive  videodisc  instruction  over 
costs  for  conventional  instruction)  found  for  this  review  were  less  than  1.0,  indicating 
lower  costs  in  every  measured  instance  for  interactive  videodisc  instruction.  The  average 
across  all  13  cost  ratios  was  0.36.  The  average  amount  of  student  time  saved  across  the 
studies  covered  by  this  review  was  31  percent  suggesting  another  source  of  savings  from 
use  of  interactive  videodisc  instruction.  Reductions  in  total  military  manpower  that  can 
result  from  such  time  savings  in  training  may  be  an  additional,  significant  source  of  cost 
savings,  but  this  possibility  was  not  examined  here. 

3.  Cost-Effectiveness 

interactive  videodisc  instruction  was  more  cost-effective  than 
conventional  instruction.  Interactive  videodisc  instruction  was  found  to  be  both  less 
costly  in  all  studies  that  reported  costs  and  more  effective  overall  than  the  conventional 
approaches  with  which  it  was  compared.  This  finding  should  be  viewed  as  suggestive 
rather  than  conclusive  since  no  studies  were  found  that  used  systematic  models  to  provide 
empirical  data  on  both  cost  inputs  and  effectiveness  outputs. 

4  .  Time  on  Task 

Interactive  videodisc  instruction  may  increase  time  on  task.  Only  one 
study  reported  an  observation  relative  to  this  point.  This  study  reported  a  45  percent 
increase  in  the  time  spent  practicing  a  targeted  task  as  a  result  of  the  introduction  of 
interactive  videodisc  instruction.  This  result  was  viewed  as  promising,  but  net  conclusive. 

5.  Retention 

Interactive  videodisc  inst  .on  seems  unlikely  to  effect  retention. 

Four  studies  addressed  retention.  Both  negative  and  positive  results  moved  closer  to  zero 
over  the  retention  intervals  considered  in  these  studies.  Post-training  experiences  typically 
influence  retention  of  knowledge  and  performance  in  a  powerful  manner.  Findings  for 
interactive  videodisc  training  corroborate  other  Defense  experiences  with  new  training 


approaches  that  suggest  their  value  lies  primarily  in  improving  the  efficiency  of  instruction, 
not  in  assuring  longer  retention. 

D.  CONCLUSION 

The  47  studies  reviewed  here  indicate  that  interactive  videodisc  instruction  is  both 
more  effective  and  less  costly  than  conventional  instruction.  They  suggest,  therefore,  that 
interactive  videodisc  instruction  can  have  a  significant  positive  impact  on  the  productivity  of 
resources  allocated  to  military  training  and  education  and  on  the  availability  of  people 
needed  to  operate  and  maintain  military  systems.  Although  more  needs  to  be  learned  about 
how  interactive  videodisc  instruction  should  be  designed  and  employed,  it  should  now  be 
routinely  considered  and  used  in  military  training  and  education. 


I.  INTRODUCTION 


A.  PURPOSE 

The  1989  Department  of  Defense  Appropriations  Bill  directed  the  Department  of 
Defense  (DoD)  to  conduct  a  study  on  the  use  of  interactive  videodisc  technology  in  training 
and  education  as  it  pertains  to  effectiveness,  cost-effectiveness,  time  on  task,  retention,  and 
overall  applicability  to  current  and  future  DoD  training  and  education  requirements. 

DoD  has  been  a  leader  in  the  development  of  interactive  videodisc  technology  and 
has  completed  many  evaluative  studies  of  its  applications  in  instruction.  For  this  reason,  a 
quantitatively-oriented,  analytic  review-a  "meta-analysis"-of  what  has  been  learned  about 
the  effectiveness  and  cost  of  interactive  videodisc  applications  in  military  training,  industrial 
training,  and  higher  education  was  undertaken  and  completed  in  response  to  the 
Congressional  direction. 

B.  BACKGROUND 
1.  Military  Training 

Military  training  in  the  United  States  is  a  substantial  enterprise.  Some  notion  of  the 
extent  and  scope  of  this  activity  is  provided  each  year  by  the  publication  of  the  Military 
Manpower  Training  Report.  This  report  describes  how  the  trained  manpower  needs 
projected  by  another  annual  report,  the  Manpower  Requirements  Report,  are  to  be  provided 
in  the  coming  fiscal  year. 

The  Fiscal  Year  1990  Military  Manpower  Training  Report  (MMTR,  1989)  reports 
that  on  an  average  day  in  FY  90  about  208,000  active  duty  personnel  and  47,000  National 
Guardsmen  and  Reservists  will  undergo  some  form  of  formal  training.  This  training  load 
increased  by  about  8  percent  in  the  1980-1990  period.  The  cost  of  this  effort  will  exceed 
$18.3  billion  in  FY  90,  and  it  will  require  the  support  of  about  176,000  military  and 
civilian  personnel  *  >  provide  instruction,  administration,  and  student  supervision. 

However,  the  MMTR  understates  the  magnitude  of  the  total  military  training 
enterprise.  It  only  concerns  instruction  conducted  in  formal  courses  by  organizations 
whose  primary  mission  is  education  or  training.  It  excludes  job-site  training,  factory  and 
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unit  training  for  new  systems,  organized  team  training  for  performance  of  specific  military 
missions,  and  all  field  exercises.  The  magnitude  of  resources  allocated  to  these  latter 
activities  is  difficult  to  determine,  but  it  is  likely  to  exceed  the  MMTR  dollar  projections  by 
a  factor  of  two  or  three. 

Military  training  is  a  complex  as  well  as  a  sizable  undertaking.  At  least  six  factors 
contribute  to  this  complexity: 

(1)  The  supply  of  people  available  for  military  service  is  decreasing.  In  the  period 
1978-1990,  the  number  of  people  reaching  age  18  each  year  dropped  from 
about  4,250,000  to  about  3,150,000.  At  present  the  Services  must  recruit 
50  percent  of  all  male  citizens  who  are  not  in  college,  not  disqualified  for 
mental,  moral,  or  physical  reasons,  and  are  age  18-21;  by  1995,  this 
proportion  could  grow  to  77  percent  (Binkin,  1986).  It  will  be  particularly 
difficult  to  recruit  people  for  high-skill  military  occupations  since  educational 
and/or  second  language  difficulties  will  become  increasingly  common  in  the 
population  of  militarily  eligible  people. 

(2)  The  number  of  military  systems  is  increasing.  At  the  end  of  World  War  I  there 
were  about  500  materiel  systems  fielded  by  the  the  U.S.  military,  at  the  end  of 
World  War  II  this  number  had  increased  to  about  2,000,  and  currently  there  are 
about  4,000  systems  either  fielded  or  in  planning.  In  the  Army  there  are  about 
0.78  large  systems  per  person--there  is  one  wheeled  vehicle  for  every  four 
people,  one  tracked  vehicle  for  every  20  people,  one  radio  for  every  six 
people,  one  generator  for  every  10  people,  and  so  on.  If  no  technological 
changes  were  made  in  the  complexity  of  military  systems,  their  quantity  and 
variety  would  by  themselves  substantially  increase  the  demands  on  military 
training  to  provide  the  people  needed  to  operate  and  maintain  them. 

(3)  The  technological  complexity  of  military  systems  is  increasing.  In  1939,  the 
volume  of  technical  documentation  required  for  the  J-F  Goose,  "Catalina 
Flying  Boat"  was  525  pages  of  information.  In  1962,  the  volume  required  by 
the  A-6A  Intruder  was  about  150,000  pages  of  information.  In  1975,  the 
volume  required  for  the  F-14  Tomcat  was  about  380,000  pages  of  information. 
In  1979,  estimates  for  the  B-l  bomber  were  in  the  neighborhood  of  1,000,000 
pages  of  information  (Kline  and  Gunning,  1979).  This  trend  will  doubtless 
continue.  Whether  the  complexity  of  military  systems  necessarily  translates 
into  increased  job  complexity  is  a  topic  of  continuing  debate.  Nonetheless,  the 
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demand  for  people  prepared  to  hold  jobs  classified  as  technical  or  highly 
technical  has  increased.  The  Army  projected  the  number  of  people  required  in 
jobs  categorized  as  "Very  Technical"  and  "Technical"  to  increase  by  about  6 
percent  from  1984-1990,  and  the  Navy  projected  an  increase  of  about  14 
percent  in  "Highly  Technical"  and  "Technical"  categories  over  the  same  period 
(Binkin,  1986). 

(4)  Costs  to  conduct  training  in  the  field  have  risen  in  absolute  terms  and  in  terms 
relative  to  other  DoD  expenses.  Fuel  and  ammunition  have  been  major 
contributors  to  increased  military  training  costs.  Fuel  costs  will  never  return  to 
pre-1970  levels,  and  ammunition  costs  continue  to  rise  along  with  the 
sophistication  of  new  military  systems.  Additionally,  the  ranges  needed  to 
exercise  the  long  reach  of  the  newest  systems  are  scarce,  expensive, 
environmentally  controversial,  and  have  increased  training  costs  substantially. 

(5)  Modem  warfare  doctrine  calls  for  more  widely  dispersed  and  more  autonomous 
units.  Access  to  training  devices,  subject  matter  expertise,  qualified 
instructors,  and  training  equipment  will  be  substantially  reduced  by  the  need  to 
maintain  unit  dispersion.  Military'  trainers  must  increase  their  capacities  for 
delivering  training  to  dispersed  and  isolated  job-sites-they  must  deliver 
training  to  students  rather  than  deliver  students  to  training. 

(6)  Reserve  component  training  poses  particularly  difficult  requirements.  The  role 
of  the  Reserve  Forces  in  the  total  force  clearly  shows  that  the  Reserves  are  no 
longer  in  reserve.  They  will  be  among  the  first  forces  committed  to  battle  in 
any  future  war.  However,  the  Reserve  components  have  a  limited  amount  of 
time  to  train,  units  are  widely  dispersed  throughout  the  country,  units  are  not 
fully  equipped,  and  only  a  small  full-time  force  of  qualified  supervisors  and 
trainers  exists  in  Reserve  units. 

Military  trainers  are  aware  of  these  challenges,  and  they  have  found  that 
conventional  approaches  will  not  meet  the  new  demands  for  increased  training  effectiveness 
without  raising  costs.  Accordingly,  they  have  sought  new  instructional  approaches  for 
meeting  these  new  requirements.  It  is  not  surprising  to  find  them  turning  to  our  most 
powerful  new  technologies  foi  these  approaches.  Interactive  videodisc  technology  is 
prominent  among  the  new  instructional  approaches  being  tried. 
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2 .  Videodisc  Technology 

Like  computer-based  instruction,  interactive  videodisc  instruction  requires  students 
to  participate  actively  in  the  training  environment,  it  provides  practice  with  high  quality 
simulation  of  devices  that  cannot  be  made  available  for  training,  and  it  distributes  and 
standardizes  both  the  content  and  interactions  of  high  quality  training  to  widely  dispersed 
sites.  Unlike  computer-based  instruction,  interactive  videodisc  instruction  also  provides 
rapid,  random  access  to  a  large,  inexpensively  stored  data  base  of  video  quality  images  and 
sequences. 

Interactive  videodisc  technology  links  videodiscs  and  videodisc  players  with 
computers.  Generally,  the  videodisc  serves  as  a  storage  medium  for  the  curriculum  data 
base,  and  the  computer  controls  how  and  in  what  order  the  curriculum  material  is  presented 
to  students.  In  this  sense,  interactive  videodisc  instruction  is  a  form  of  computer  based 
instruction,  but  the  power  and  functionalities  added  by  videodisc  technology  give  it  an 
identity  of  its  own. 

Videodisc  technology  uses  metalized  plastic  discs  to  code  information  as  tiny  pits 
(about  0.5  microns  in  diameter)  pressed  into  a  transparent  substrate  with  a  reflective 
coating.  During  play  of  the  disc,  the  pits  modulate  a  laser  signal  that  is  decoded  into  audio 
and  video  signals.  About  15  companies  possess  manufacturing  methods  and  processes  to 
master  and  replicate  such  discs  from  a  master  videotape  submitted  by  a  curriculum 
developer.  The  first  copy  of  a  disc  may  cost  $2,000-$3,000,  but  additional  copies  may 
cost  less  than  $20  depending  on  quantity.  DoD  has  specified  a  standard  format, 
LaserVision,  for  production  of  all  the  videodiscs  it  uses. 

Videodiscs  are  available  in  several  sizes-the  most  common  are  8  inches  and  12 
inches  in  diameter.  The  12  inch  interactive  videodiscs  can  store  30  minutes  of  full-color 
video,  or  54,000  video  frames,  on  each  disc  side  along  with  two  tracks  of  audio.  With 
audio  compression,  150  hours  of  audio  can  be  stored  on  each  disc  side.  Videodiscs  can 
store  digital  as  well  as  analog  data.  Depending  on  size  and  format  they  can  store  200-3000 
MB  (megabytes)  of  information.  An  interactive  videodisc  system  is  an  audiovisual 
instructional  system  using  a  "personal"  microcomputer,  videodisc  player,  monitor,  and 
special  interfaces  for  displaying  graphics  and  controlling  the  player.  Such  a  system  will 
generally  cost  $4,500-$8,000.  The  videodisc  player  and  its  interface  typically  adds  about 
$2,500  to  a  personal  computer  system, 
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Levels  of  interactivity--or  the  ability  to  respond  to  the  actions  of  users-have  been 
identified  for  videodisc  systems  of  different  configurations.  Supplying  trainers  with  a 
videodisc  system  at  one  level  or  another  does  not  guarantee  that  it  will  be  used  in  a  fashion 
commensurate  with  its  potential.  This  potential  may  or  may  not  be  fully  used  depending  on 
the  imagination  and  capabilities  of  instructional  developers.  However,  these  levels  do 
identify  progressively  higher  ceilings  for  interactivity. 

The  following  definitions  of  videodisc  levels  of  interactivity  were  developed  by  the 
Joint  Service  Ad  Hoc  Committee  for  Interactive  Courseware  Data  Item  Descriptions.  This 
committee  based  its  definitions  on  those  developed  by  the  University  of  Nebraska 
Videodisc  Design  and  Production  Group,  which  in  turn  were  derived  from  common 
developer  usage.  They  are  the  following: 

•  Level  0— A  videodisc  system  intended  for  linear  play  without  interruption. 

•  Level  I-*A  videodisc  system  with  still/freeze  frame,  picture  stop,  frame  and 
chapter  search,  dual  channel  audio,  but  no  programmable  memory.  All 
functions  are  intended  to  be  initiated  by  manual  inputs  from  the  videodisc 
player's  keypad.  Picture  stop  and  chapter  stop  are  read  from  the  videodisc. 

•  Level  II— A  videodisc  system  with  on-board,  programmable  memory.  The 
videodisc  player's  memory  is  programmed  by  "digital  dumps"  from  audio 
channel  two  of  ihe  videodisc  or  by  manual  entry  from  the  videodisc  player's 
keypad.  Inputs  are  made  from  the  keypad  or  from  a  device  that  emulates  the 
keypad. 

•  Level  HI— A  videodisc  system  in  which  the.  videodisc  player  is  interfaced  to  an 
external  computer.  The  videodisc  player  acts  as  a  computer  peripheral  with  its 
functions  under  the  computer's  control. 

•  Level  IV— A  videodisc  system  in  which  the  videodisc  player  is  interfaced  to  an 
external  computer.  The  videodisc  functions  both  as  an  optical  storage  device 
for  digital  information  and  as  the  source  of  analog  picture  and  sound.  The 
video  frames  on  the  videodisc  store  digital  data  intended  to  be  read  and 
processed  by  the  computer. 

Since  the  difference  between  Level  III  and  Level  IV  depends  on  where  and  how 
digital  information  is  stored-in  Level  III  it  is  generally  stored  on  a  digital  device,  such  as  a 
magnetic  disk,  and  in  Level  IV  it  is  stored  on  videodisc-whether  or  not  Level  IV  provides 
the  potential  for  more  interactivity  than  does  Level  III  is  a  topic  of  continuing  debate. 


The  five  definitions  are  listed  here  for  the  record.  Only  two  levels  (II  and  III)  were 
found  among  the  evaluations  identified  for  this  review.  The  term  "interactive  videodisc"  in 
contrast  with  "videodisc"  alone  implies  the  presence  of  a  computer,  and  it  is  used  here  to 
refer  to  Level  IH  presentations. 

3 .  Interactive  Videodisc  Instruction 

New  optical  technologies  based  on  compact  discs  such  as  CD-ROM  (compact  disc- 
read  only  memory),  CD-ROM/XA  (CD-ROM  Extended  Audio),  CD-I  (compact  disc- 
interactive),  DVI  (digital  video-interactive),  and  others  still  emerging  (e.g.,  UVC 
Corporation's  approach  to  video  compression),  as  well  as  new  approaches  for  computer¬ 
generated  imagery,  will  compete  for  the  market  currently  held  by  videodisc  systems. 
However,  the  instructional  capabilities,  or  "functionalities,"  of  interactive  videodisc 
systems  are  the  concern  of  this  review.  Regardless  of  the  hardware  system  eventually 
chosen  to  deliver  them,  the  functionalities  made  available  by  interactive  videodisc 
technology  are  likely  to  be  preserved  and  pursued  by  the  training  community  for  the 
foreseeable  future. 

These  functionalities  include  those  seen  in  computer-based  instruction.  Drill  and 
practice,  tutorials,  simulation,  and  gaming  may  all  be  enhanced  by  the  ability  of  videodiscs 
to  store  on  one  randomly  accessible  medium  substantial  amounts  of  information  in  many 
different  formats  such  as  video  still  images,  motion  video,  analog  audio  (on  separate  and 
different  tracks),  and  digital  information  such  as  compressed  audio,  curriculum  data  bases, 
and  computer  programming. 

Much  of  the  development  of  instructional  functionalities  now  found  in  both 
computer  based  instruction  and  interactive  videodisc  technology  has  been  supported  by 
DoD  (Power  On!,  1988).  Fletcher  and  Rockway  (1986)  reviewed  the  history  of  these 
contributions  and  identified  five  new  functionalities  developed  by  the  Department  of 
Defense  for  interactive  videodisc  instruction.  These  five  are  the  following. 

•  Surrogate  travel.  This  capability  simulates  a  path  of  travel  selected  by  the 
user  over  »errain  representations  stored  on  videodisc.  Areas  chosen  to 
demonstrate  this  capability  have  included  a  small  city,  a  harbor,  a  nuclear 
power  plant,  and  an  art  gallery.  Using  surrogate  travel,  the  user  chooses  the 
path,  controls  the  speed  of  advance,  and  directs  the  angle  of  view  while 
"traveling"  through  an  area  using  simple  controls-usually  a  joystick.  When 
the  user  comes  to  a  choice  point  such  as  an  intersection,  he/she  can  turn  right, 
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turn  left,  proceed  ahead,  or  go  back.  The  videodisc  frames  that  the  user  sees 
originate  as  filmed  views  of  what  one  would  actually  see  in  the  area-possibly 
including  what  one  would  see  at  different  times  of  day  or  at  different  seasons 
of  the  year. 

•  Interactive  movies.  One  problem  with  training  that  requires  demonstrations 
of  skilled  performance  is  that  essential  components  of  the  demonstrations  may 
be  invisible  to  viewers.  Interactive  movies  are  intended  to  solve  this  problem 
by  allowing  the  user  interactive  control  over  many  aspects  of  viewing  such  as 
direction  (front,  side,  above,  below),  speed  (fast,  slow,  still-frame,  reverse), 
detail  (panning  and  zooming),  abstraction  (photographs,  labeled  line-drawing 
animations),  plot  (different  actions  at  different  choice  points),  and 
simultaneous  action  (gauge  readings,  actions  outside  of  the  current  angle  of 
view).  Interactive  movies  are  prepared  by  storing  many  different  views  of 
skilled  performance  on  videodisc  and  making  them  available  in  parallel  under 
the  viewer's  control. 

•  Microtravel.  This  capability  combines  surrogate  travel  with  interactive 
movies  and  allows  travel  in  places  where  people  cannot  go.  One 
demonstration  of  this  capability  for  Defense  training  provided  microtravel 
through  a  jeep  engine  while  it  was  running. 

•  Spatial  data  management.  Electronic  libraries  in  the  form  of  spatial  data 
management  systems  allow  users  to  "fly-over"  spatially  organized  data  using 
joystick  control. .  Data  elements  are  associated  with  familiar  terrain  (e.g.,  a 
university  campus)  so  that  anyone  familiar  with  the  terrain  can  locate  data  of 
interest-information  on  chemistry  would  be  found  by  "flying"  to  the  chemistry 
department,  information  on  tuba  concertos  might  be  found  by  "flying"  to  the 
tuba  section  of  the  university  symphony  orchestra  found  in  the  music 
department,  and  so  on.  The  data  may  be  in  almost  any  form,  and  the  spatial 
data  management  systems  developed  by  DoD  provided  the  earliest 
implementations  and  demonstrations  of  this  approach  to  hypermedia 
presentations. 

•  Low-cost  portable  simulators.  One  of  the  first  military  skill  training 
videodiscs,  developed  in  the  late  1970s,  was  an  Army  tank  gunnery  trainer. 
This  interactive  videodisc  simulation  i  ;es  motion  video  segments  stored  on 
videodisc  to  provide  tactical  environments  and  targets.  The  student  selects 
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ammunition,  aims,  and  fires  using  computer  generated  imagery  overlaid  onto 
the  video  display  to  receive  round  trajectory  and  burst  on  target  feedback.  As 
can  be  seen  from  the  evaluations  discussed  in  this  report,  many  other 
simulations  were  subsequendy  developed  for  military  training. 

In  Defense  as  elsewhere,  training  is  a  major  application  for  interactive  videodisc 
technology.  About  half  of  the  approximately  100,000  videodisc  systems  in  non-consumer 
use  during  the  mid-1980s  in  the  United  States  were  used  for  training  (Miller,  1987).  The 
effectiveness  of  these  applications  is  of  substantial  and  natural  interest  both  inside  and 
outside  DoD.  Many  organizauons  want  to  know  what  their  investment  in  interactive 
videodisc  instruction  has  bought  in  terms  of  increased  instructional  effectiveness.  That  is 
the  principal  topic  of  the  review  reported  here. 


II.  APPROACH 


This  review  is  limited  to  an  instructional  medium-interactive  videodisc.  Therefore 
one  criterion  for  including  studies  in  this  review  was  the  use  of  a  videodisc  hardware 
system.  The  unambiguity  of  this  criterion  clarifies  what  studies  were  candidates  for 
inclusion  in  this  review,  it  aids  replication,  and  it  helps  readers  judge  for  themselves  the 
relevance  of  this  work  to  their  own  needs. 

On  the  other  hand,  the  approach  is  subject  to  the  cautions  raised  by  many 
commentators  and  best  articulated  by  Clark's  (1983)  critique  of  research  on  the 
instructional  effectiveness  of  media.  These  concerns  may  be  summed  up  by  the  notion  that 
hardware  alone  does  not  define  an  instructional  approach-what  is  done  with  the  hardware 
is  what  counts.  This  point  of  view  seems  unequivocal.  The  presence  of  a  computer 
controlled  videodisc  system  is  no  guarantee  of  effective  instruction  nor  even  that  the  unique 
features  of  the  system  will  be  used. 

Critics  of  media  effectiveness  research  push  the  argument  farther.  Clark  states  that 
"The  best  current  evidence  is  that  media  are  mere  vehicles  that  deliver  instruction  but  do  not 
influence  student  achievement  any  more  than  the  truck  that  delivers  our  groceries  causes 
changes  in  our  nutrition"  (page  445).  This  statement  may  go  too  far.  Improvements  in  the 
technology  of  delivering  food  from  centers  of  production  to  markets  has  had  a  tremendous 
impact  on  the  nutrition  of  nations.  The  technologies  by  themselves  do  not  guarantee  this 
impact,  but  the  functionalities  they  support  and  their  applications  do.  The  development  of 
movable  type  did  not  by  itself  have  an  impact  on  learning,  but  the  functionalities  it 
supported  and  their  application  to  instruction  did-no  books,  no  mass  education. 

This  argument  may  extend  to  any  medium,  interactive  videodisc  included.  The 
functionalities  made  possible  and  affordable  by  interactive  videodisc  technology  hold 
substantial  promise  for  instruction.  Experimental  investigation  of  these  functionalities  is 
essential  to  their  effective  application,  and  work  concerning  interactive  videodisc 
functionalities  such  as  that  being  done  by  Hannafin  and  his  associates  (e.g.,  Hannafin, 
1985;  Phillips,  Hannafin,  and  Tripp,  1988)  and  others  (e.g.,  Gay,  1986)  -even  when  it 
employs  videotape  rather  than  videodisc  (e.g.,  Levenson,  Morrow,  and  Signer,  1985; 
Tiene,  Evans,  and  Milheim,  1988) — is  needed,  commendable,  and  to  be  encouraged,  as 


Clark's  article  suggests.  This  review,  however,  is  more  summative.  It  is  intended  to  bring 
together  what  we  have  learned  thus  far  about  the  effectiveness  of  interactive  videodisc 
technology  applied  to  instruction. 

The  following  criteria  guided  the  choice  of  studies  for  this  review. 

(1)  The  studies  had  to  conform  to  standard  criteria  for  care  and  control  in  the 
conduct  of  evaluation  research.  These  criteria  are  generally  accepted  and 
discussed  in  research  literature  concerning  methodology  for  the  design  and 
conduct  of  experiments  in  the  social  and  behavioral  sciences. 

(2)  The  studies  had  to  involve  a  comparison  of  an  instructional  approach  using 
interactive  videodisc  with  conventional  instruction.  "Conventional  instruction" 
was  assumed  to  involve  all  of  the  non-or  limited-interaction  approaches  now 
available  to  instructors  including  platform  lecture,  text,  programmed  text,  on 
the  job  training,  experience  with  actual  equipment,  and  even  video  and 
videotape  instruction.  It  excluded  highly  interactive  approaches  such  as 
computer-based  instruction  or  interactive  videodisc  instruction. 

This  requirement  meant  that  the  choice  of  studies  was  limited  to 
comparisons  of  experimental  and  control  group  performance.  It  is  possible, 
and  perhaps  desirable,  to  evaluate  an  instructional  approach  without  these 
comparisons.  For  instance,  tracing  a  "trajectory"  of  student  progress  through  a 
course  as  described  and  applied  by  Suppes,  Fletcher,  and  Zanotti  (1976) 
directly  relates  time  in  instruction  to  achievement  and  is  more  powerful  than  the 
experimental  group/control  group  comparisons  sought  here.  However, 
comparisons  were  needed  to  calculate  effect  size,  and  studies  without  them 
were  excluded.  Some  studies  that  only  examined  the  effectiveness  of  different 
approaches  to  the  design  of  videodisc  instruction  were  excluded.  Those  that 
compared  different  levels  of  interactivity  were  included  in  this  review,  but 
others  of  this  sort  were  not. 

(3)  The  studies  had  to  involve  computer-controlled-Level  Ill-videodiscs.  Four 
studies  were  found  that  involved  Level  II  videodiscs.  These  are  reported  and 
discussed  briefly,  but  their  results  are  not  aggregated  with  the  descriptive 
statistics  reported  for  Level  III  studies.  Some  good  experimental  studies 
turned  up  (e.g.,  Levenson,  Morrow,  and  Signer,  1985)  that  used  videotape 
players  to  mimic  the  interactive  capabilities  of  instruction  using  videodisc 
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players.  However,  they  did  not  use  videodiscs,  and  they  were  not  included 
among  the  studies  reviewed  here. 

(4)  The  studies  had  to  report  sufficient  data  to  permit  calculation  of  effect  size. 
Generally,  means  and  standard  deviations  were  needed  for  all  treatment 
groups.  In  the  absence  of  these  data  it  was  occasionally  possible  to  estimate 
effect  size  from  other  information  provided-especially  information  obtained 
from  significance  tests.  Glass,  McGaw,  and  Smith  (1981)  describe  a  number 
of  ways  to  calculate  effect  size  when  means  and  standard  deviations  are  not 
provided  directly.  These  methods  were  used  when  they  could  be. 

(5)  The  studies  had  to  concern  military  training  or  use  students  of  about  the  same 
age  and  type  as  military  training  students.  In  practice,  this  meant  that  the 
studies  were  limited  to  military  training,  higher  education,  and  industrial 
training.  Only  one  study  using  non-handicapped  elementary  school  students 
(Hasselbring  et  al.,  1988)  and  only  two  using  non-handicapped  secondary 
school  students  (Blatnik  and  Holden,  1988;  Gale  and  Barksdale,  1989)  turned 
up. 

(6)  All  work  with  handicapped  students  was  excluded.  Research  on  videodisc 
applications  for  handicapped  students  has  a  long  and  vigorous  history.  Work 
by  Hofmeister  (e.g.,  Hofmeister,  Engelmann,  and  Caminc,  1986), 
Thorkildsen  (e.g.,  1985),  Browning  (e.g.,  Browning  et  al.,  1986),  and  their 
colleagues  is  notable.  A  review  of  this  work  could  probably  be  supported  by 
the  state  of  the  art,  but  it  is  beyond  the  scope  of  this  investigation. 

This  investigation  benefited  from  other  reviews  of  interactive  videodisc  instruction. 
These  reviews  include  those  by  Bosco  (1986),  DeBioois,  Maki,  and  Hall  (1984),  DeBloois 
(1988),  an  annotated  bibliography  compiled  by  Sarli  et  al.  (1988),  and  a  review  completed 
by  Capper  (in  press).  On  the  basis  of  these  reviews,  it  was  evident  that  the  state  of  our 
knowledge  would  support  a  quantitatively  oriented,  analytic  review  of  the  effectiveness  of 
interactive  videodisc  instruction. 

A.  TECHNICAL  APPROACH 

The  methodology  used  for  analytic  reviews  has  changed  considerably  in  the  last  15 
years.  The  "box  score"  approach  which  earlier  characterized  the  methodology  has  been 
replaced  by  "meta-analysis."  In  the  box-score  approach,  studies  in  which  an  experimental 
group  exposed  to  the  treatment  under  review  are  collected,  the  proportion  of  studies  in 
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which  the  experimental  group  means  exceed  control  group  means  by  some  statistically 
significant  extent  is  calculated,  and  the  treatment  is  reported  as  favorable  or  not  depending 
on  whether  this  proportion  is  large  or  small.  Hedges  and  Olkin  (1980)  have  shown  that  the 
box  score  approach  has  very  low  power— low  ability  to  detect  statistically  significant 
differences  between  two  gToups--for  the  treatment  effect  sizes  and  sample  sizes 
characteristic  of  social  science  research.  Tney  also  showed  that  the  power  of  the  procedure 
decreases  as  the  number  of  studies  included  in  the  review  increases. 

Glass  (1976)  proposed  an  alternative  approach.  Since  he  was  performing  an 
analysis  of  analyses,  he  described  his  alternative  as  "meta-analytic."  It  differs  from  the 
box-score  approach  in  three  ways:  (1)  studies  relevant  to  the  issue  at  hand  are  collected 
using  clearly  defined  procedures  that  can  be  replicated;  (2)  a  quantitative  measure,  "effect 
size,"  is  used  to  tabulate  the  outcomes  of  all  the  collected  studies  including  those  with 
results  that  are  not  statistically  significant;  and  (3)  statistical  procedures  are  used  to 
synthesize  the  quantitative  measures  and  describe  the  findings  of  the  analysis.  Glass’s 
approach  appears  to  be  especially  appropriate  for  synthesizing  the  results  of  instructional 
research,  and  it  has  been  widely  used  for  this  purpose  since  its  introduction. 

Meta-analysis  is  still  developing  as  a  technique  and  some  matters  concerning  its  use 
remain  unsettled.  The  issues  receiving  the  most  attention  at  present  concern  the  "fiie- 
drawer  problem"  and  the  calculation  of  effect  size.  The  use  of  meta-analysis  to  perform 
research  syntheses  that  include  costs  (syntheses  involving  cost-effectiveness)  remains 
unaddressed  and  undetermined. 

The  file-drawer  problem  may  stem  from  the  reluctance  of  professional  journal 
editors  to  include  studies  whose  results  fail  to  reach  statistical  significance  and  the 
reluctance  of  researchers  to  report  their  null  results-these  studies,  therefore,  remain  in  file- 
drawers.  The  question,  then,  is  how  much  effect  would  these  inaccessible  studies  have  on 
the  results  of  our  meta-analyses?  After  some  analysis,  the  answer  according  to  Rosenthal 
(1984)-among  others-seems  to  be  not  much.  Nonetheless,  steps  were  taken  in  this 
review  to  locate  studies  left  in  file-drawers. 

Effect  size  is  usually  defined  as  the  difference  between  the  means  of  two  groups 
divided  by  the  standard  deviation  of  the  control  group.  Effect  sizes  calculated  in  this  way 
estimate  the  difference  between  two  gioup  means  measured  in  control  group  standard 
deviations.  Glass  et  al.  (1981)  suggest  that  choice  of  the  denominator  is  critical  and  that 
choices  other  than  the  control  group  standard  deviation  are  defensible.  However,  they 
endorse  the  standard  choice:  using  the  control  group  standard  deviation. 
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Alternatively,  Hedges  and  Olkin  (1985)  show  that,  for  every  effect  size,  both  the 
bias  and  variance  of  its  estimate  are  smaller  when  the  standard  deviation  is  obtained  by 
pooling  the  sample  variance  of  two  groups  instead  of  using  the  control  group  standard 
deviation  by  itself.  An  effect  size  based  on  a  pooled  standard  deviation  estimates  the 
difference  between  two  group  means  measured  in  standard  deviations  estimated  for  the  full 
population  from  which  both  experimental  and  control  groups  are  drawn.  Both  approaches 
are  used  in  this  review  to  estimate  effect  sizes  and  both  are  reported  whenever  sufficient 
data  were  available  to  calculate  them. 

If  the  standard  deviations  for  both  control  and  experimental  groups  are  about  the 
same  in  magnitude,  effect  sizes  calculated  using  control  group  standard  deviations  will 
differ  little  from  those  calculated  using  pooled  standard  deviations.  If,  on  the  other  hand, 
effect  sizes  are  larger  when  they  are  calculated  using  pooled  standard  deviations  than  they 
are  when  they  are  calculated  using  control  group  standard  deviations,  there  must 
necessarily  be  smaller  standard  deviations  and  less  variability  across  the  measure  of  interest 
in  the  experimental  group  than  in  the  control  group.  This  is  the  case  found  here.  The  effect 
sizes  calculated  in  this  review  using  pooled  standard  deviations  were  generally  larger  than 
those  calculated  using  control  group  standard  deviations. 

Most  commentators  suggest  that  effect  sizes  can  be  treated  as  descriptive  statis  cs 
and  entered  into  standard  tests  for  statistical  significance.  Hedges  and  Olkin  (1985)  have 
shown  that  the  error  variance  around  estimates  of  effect  size  is  inversely  proportional  to  the 
sample  size  of  the  studies  from  which  the  effect  sizes  are  drawn.  If  the  effect  sizes  in  any 
review  are  drawn  from  studies  employing  widely  different  sample  sizes,  then  the 
heterogeneity  of  variance  among  effect  sizes  prohibits  their  use  in  conventional  t-tests, 
analyses  of  variance,  etc.  In  this  review  the  sample  sizes  vary  by  a  factor  of  20;  hence  the 
effect  sizes  reported  here  are  treated  only  with  descriptive  statistics.  Standard  deviations 
and  sample  sizes  for  effect  sizes  are  reported  so  that  others  may  examine  the  results  further 
using  such  inferential  means  as  they  wish. 

B .  DATA  COLLECTION 

Defense  Technical  Information  Center,  Educational  Resources  Information  Center, 
and  Psychological  Abstracts  data  bases  were  searched  using  all  combinations  of  the 
following; 

[Computer  <or>  videodisc] 

<and> 


[Assisted  <or>  Aided  <or>  Mediated  <or>  Managed  <or>  Based  <or>  <Empty>] 

<and> 

[Education  <or>  Learning  <or>  Training] 


Additionally,  the  following  terms  were  used  by  themselves: 

•  Interactive  Videodisc 

•  Interactive  Video 

•  Interactive  Courseware. 

This  search  turned  up  more  than  2,400  candidate  studies.  Most  of  these  studies 
concerned  computer-based  instruction  alone  and  not  interactive  videodisc  instruction.  This 
search  was  not  without  value,  since  it  identified  about  two-thirds  of  the  studies  eventually 
used  in  this  review.  On  the  other  hand,  there  were  several  studies  that  were  known  to  exist 
but  that  did  not  turn  up  in  this  search.  Some  of  these  studies  are  unpublished  and  not 
catalogued  anywhere.  Others  are  published  but  not  catalogued  using  any  of  the  above 
keywords.  Still  others  exist  only  as  laboratory  reports.  Whatever  the  case,  it  was 
necessary  to  decide  whether  or  not  to  include  studies  that  did  not  turn  up  in  the  formal 
search  process.  A  decision  was  made  to  include  them. 

To  locate  uncatalogued,  unpublished  evaluation  studies,  we1  contacted  cognizant 
individuals  in  the  training  commands  of  all  three  Services  (US  Army  Training  and  Doctrine 
Command,  US  Navy  Chief  of  Education  and  Training,  and  US  Air  Force  Air  Training 
Command),  in  the  personnel  research  and  development  commands  of  all  three  Services 
(US  Army  Research  Institute,  US  Navy  Personnel  Research  and  Development  Center,  and 
US  Air  Force  Human  Resources  Laboratory,  among  others),  and  in  specific  Service 
schools  where  we  knew  interactive  videodisc  materials  had  been  used. 

We  also  contacted  industrial  training  organizations  to  see  if  they  had  evaluation  data 
and  if  they  were  willing  to  release  it.  We  pursued  this  by  publishing  a  notice  in  the 
Videodisc  Monitor,  by  calling  every  developer  listed  in  the  March  1988  InfoWindow 
Courseware  Catalog-,  and  by  calling  every  developer  listed  in  the  1988-1989  Videodisc 
Compendium  for  Education  and  Training.  With  these  actions  we  hoped  to  unearth 
"fugitive"  documents  that  had  reported  studies  performed  for  business  but  not  formally 


1  was  assisted  in  this  process  by  Dr.  Ruth  Wienclaw,  to  whom  I  am  grateful. 
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published.  Few  such  documents  emerged.  We  were  gratified  by  the  willingness  of 
researchers  to  root  through  their  personal  files  and  dig  out  references  that  were  already  in 
the  published  literature.  Some  of  these  references  had  not  been  identified  by  our  search  of 
the  data  bases,  and  we  were  glad  to  know  about  them. 

Finally,  we  telephoned  a  number  of  researchers  directly  to  obtain  original  data  to 
permit  calculation  of  effect  sizes  when  the  necessary  data  were  missing  from  their  report 
documentation.  As  usual  we  encountered  helpful,  facilitating  people  who  were  willing  to 
dig  through  their  personal  files  and  send  us  whatever  they  felt  would  be  relevant. 


III.  FINDINGS  AND  DISCUSSION 


# 


A.  TABULATION  OF  STUDIES 

Tht  effectiveness  of  interactive  videodisc  applications  in  training  and  education  can 
be  assessed  by  examining  the  effect  sizes  reported  in  Appendix  A.  That  appendix  is 
intended  to  invite  exploration  of  data  collected  for  this  review  and  (re)examination  of  the 
findings  presented  in  this  paper. 

Outside  of  what  could  be  determined  by  descriptions  of  the  instructional  approach 
and  what  could  be  determined  from  the  evaluation  results,  direct  assessment  of  interactive 
videodisc  content  or  quality  was  beyond  the  scope  and  resources  of  this  investigation. 
Aspects  such  as  the  quality  of  graphics,  clarity  of  instructional  text,  and  verisimilitude  of 
simulations  were  not  assessed.  The  state  of  the  art  in  interactive  videodisc  instruction  could 
most  probably  support  such  an  analysis,  and  it  would  be  an  appropriate  next  step,  but  it 
was  not  attempted  here. 

There  are  28  studies  summarized  in  Appendix  A— some  with  more  than  one 
reference.  The  Appendix  is  organized  so  that  the  13  military  training  studies  are  listed  first, 
the  4  industrial  training  studies  are  listed  next,  and  the  1 1  higher  education  studies  are  listed 
last.  Within  these  groups,  the  studies  are  listed  chronologically.  Two  studies  were 
performed  at  the  US  Air  Force  Academy  (Crotty,  1984;  Verano,  1987).  Although  these 
studies  used  military  subjects  (cadets),  their  setting  seemed  more  characteristic  of  higher 
education  than  military  training,  and  they  are  included  with  the  higher  education  studies. 

Although  9  of  the  17  studies  in  military  and  industrial  training  settings  concerned 
electronic  maintenance,  instructional  content  concerned  a  variety  of  subject  matter, 
including  "softer"  topics  such  as  military  operations  in  urban  terrain  (King  and  Reeves, 
1985),  interpersonal  skills  (Schroeder  et  al.,  1986),  and  handling  of  hazardous  materials 
(Bosco  and  Wagner,  1988).  None  of  the  1 1  studies  in  higher  education  settings  concerned 
electronics  maintenance.  Their  instructional  content  ranged  from  equipment  operation 
(Fowler,  1980),  CPR  instruction  (Lyness,  1987),  and  medical  education  (Allan,  1989) 
through  biology  (Bunderson,  et  al.,  1984,  Davis  1985),  physics  (Stevens,  1984),  and 
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chemistry  (Smith  et  al.,  1986;  Jones,  1987;  Jones  1988),  to  foreign  language  instruction 
(Crotty,  1984;  Verano,  1987). 

The  main  distinction  in  instructional  approaches  was  among  those  applications  that 
used  the  videodisc  as  a  tutorial,  programmed  textbook  with  visuals— the  intrinsic 
programming  approach  advocated  by  Crowder  (1959)-,  those  that  used  the  videodisc  to 
permit  free  exploration  of  simulated  situations  or  equipment,  and  those  that  used  both  by 
combining  tutorials  with  simulations.  Judgments  concerning  instructional  approach  were 
based  on  documentation  of  the  instruction  and  on  discussions  with  developers  or  others 
who  had  direct  experience  with  it. 

In  Table  A-l  (Appendix  A),  the  "N"  column  lists  the  number  of  subjects  in  the 
experimental  (Exp)  and  the  control  (Ctrl)  groups. 

The  Comparison  column  tells  what  the  experimental  and  control  treatments  were. 
Generally  the  experimental  group  involved  interactive  videodisc  (IVD)  presentations,  and 
the  control  group  did  not.  In  some  cases,  different  interactive  videodisc  presentations  were 
compared  with  each  other  or  computer  assisted  instruction  (CAI)  was  compared  with  the 
control  treatment.  Findings  from  these  comparisons  are  also  included  in  Appendix  A.  One 
paradigm  that  is  listed  frequently  in  the  Comparison  column  involves  instruction  using 
videodisc-based  simulations  of  equipment  compared  with  instruction  using  hands-on 
experience  with  the  actual  equipment  (AE)  that  was  the  object  of  the  instruction. 

As  shown  in  the  Outcome  column,  four  categories  of  effectiveness  data  for 
interactive  videodisc  instruction  were  identified.  These  categories  were:  knowledge 
outcomes,  which  assessed  students'  knowledge  of  facts  and/or  concepts  presented  by  the 
instruction;  performance  outcomes,  which  assessed  students'  skill  in  performing  some 
criterion  tasks  or  procedures;  retention,  which  reported  knowledge  or  performance 
measures  taken  some  time  after  instruction  ceased;  and  time  to  complete  the  instruction.  It 
was  assumed  that  time  to  complete  a  performance  test  was  properly  considered  as  a 
performance  outcome  measure,  but  that  time  to  complete  the  instruction  itself  should  be 
considered  separately. 

Effect  size  (ES)  was  calculated  as  the  difference  in  the  group  means  divided  both  by 
the  pooled  standard  deviation  (Comb  SD)  and,  where  possible,  by  the  control  group 
standard  deviation  (Ctrl  SD).  In  some  cases,  effect  size  was  estimated  from  other  statistics 
using  procedures  suggested  by  Glass  et  al.  (1981). 

Some  studies  involved  simultaneous  comparison  of  more  than  two  treatment  groups 
and  began  with  an  analysis  of  variance  (ANOVA)  before  comparing  the  treatment  groups  in 
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a  pairwise  manner.  The  statistical  significance  of  the  analysis  of  variance  comparison  is 
reported  in  the  table.  All  effect  sizes  reported  were  calculated  from  pairs  of  treatments. 
Each  effect  size  was  calculated  so  that  it  was  positive  if  the  difference  favored  the  videodisc 
treatment  and  negative  if  it  favored  the  control  treatment.  More  specifically,  and  to  cover 
cases  such  as  those  in  which  different  videodisc  treatments  were  compared  with  each  other, 
effect  size  was  always  calculated  so  that  it  was  positive  if  it  favored  the  instructional 
treatment  judged  to  be  the  most  interactive  of  the  pair.  The  treatment  judged  to  be  most 
interactive  is  always  the  first  of  the  two  treatments  listed  in  the  corresponding  Comparison 
column. 

The  abbreviation  "ns"  is  used  in  the  Effect  Size  column  to  indicate  that  the  observed 
difference  in  treatments  was  not  statistically  significant  (p  £  .05).  Otherwise,  the 
significance  level  reported  in  the  study  is  listed. 

Superscripts  are  used  in  the  "ES  Comb  SD"  (Effect  Sizes  Based  on  Combined  or 
Pooled  Standard  Deviations)  column  to  indicate  the  categories  into  which  the  effect  size 
was  coded.  Effect  sizes  based  on  knowledge  outcomes  are  signified  by  a  superscript  "K", 
and  those  based  on  performance  outcomes  are  signified  by  a  superscript  "P".  Effect  sizes 
based  on  time  to  complete  the  instruction  are  signified  with  a  superscript  "D"  (duration). 
Only  8  studies  in  Appendix  A  reported  time  to  complete  instruction,  so  knowledge  and 
performance  outcomes  were  combined  and  coded  together  for  these  duration  outcomes. 
Effect  sizes  based  on  retention  are  signified  with  a  superscript  "R".  Only  four  studies  in 
Appendix  A  measured  retention,  so  knowledge  and  performance  outcomes  were  also 
combined  and  coded  together  for  retention  outcomes. 

Superscripts  are  also  used  in  "ES  Comb  SD"  column  to  indicate  how  the  effect 
sizes  were  coded  with  respect  to  instructional  approach.  Effect  sizes  judged  to  be  based  on 
tutorial  approaches  without  simulation  are  signified  by  a  superscript  "T",  those  judged  to  be 
based  on  tutorial  approaches  combined  with  simulation  are  signified  by  a  superscript  "C", 
and  those  judged  to  be  based  on  simulation  alone  are  signified  by  a  superscript  "S". 

Four  studies  reported  "Baseline"  comparisons  of  interactive  videodisc  instruction 
with  control  conditions  in  which  no  instruction  or  irrelevant  placebo  instruction  was 
provided.  Generally,  these  studies  tell  us  if  interactive  videodisc  instruction  teaches 
anything.  Effect  sizes  concerning  these  baseline  comparisons  are  signified  with  a 
superscript  "B"  in  the  appendix. 

Some  studies  used  interactive  videodisc  instruction  in  both  comparison  groups  but 
compared  high  or  more  extensive  interactivity  with  low  or  less  extensive  interactivity. 
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Effect  sizes  based  on  these  comparisons  of  level  of  interactivity  are  signified  with  a 
superscript  "I". 

Some  of  the  effect  sizes  reported  in  Appendix  A  are  included  for  general  interest 
and  completeness  but  were  not  used  in  this  review.  Among  these  are  effect  sizes  based  on 
comparisons  that  did  not  involve  videodisc  instruction  at  all.  For  instance,  one  of  the  post- 
ANOVA  pairwise  comparisons  reported  by  Gibbons  et  al.  (1983)  concerned  computer- 
assisted  instruction  (CAI)  and  classroom  instruction,  and  its  effect  size  was  not  considered 
further  in  this  review.  Also  reported  but  excluded  from  further  consideration  were  effect 
sizes  involving  interactive  videodisc  instruction  in  the  experimental  group,  but  an 
interactive  approach  in  the  control  group  not  ordinarily  viewed  as  conventional  instruction. 
For  instance,  another  post-ANOVA  comparison  reported  by  Gibbons  et  al.  involved 
computer- assisted  instruction  in  the  control  group.  Another  example  is  Fowler's  (1980) 
comparison  of  linear  interactive  videodisc  instruction  with  branched  videotape  instruction. 
Finally  excluded  from  further  consideration  were  effect  sizes  involving  interactive 
videodisc  instruction  in  which  the  use  of  the  videodisc  capability  was  constrained—usually 
for  good  experimental  reasons.  For  instance,  Verano  (1987)  included  comparisons  using 
videodisc  instruction  constrained  to  strictly  iinear  presentations. 

The  signs  of  values  reported  in  the  Percent  Improvement  column  were  determined 
in  a  manner  similar  to  Jiat  used  for  the  Effect  Size  column.  Each  percent  was  calculated  so 
that  it  is  positive  when  the  difference  favored  the  most  interactive  treatment  (i.e.,  the 
treatment  listed  firsi  in  the  corresponding  Comparison  column). 

Cost  ratios  are  reported  as  the  ratio  of  videodisc  treatment  costs  over  the  control 
treatment  costs,  or,  again,  the  most  interactive  treatment  costs  over  the  other  treatment 
costs.  The  closer  the  proportion  is  to  zero,  the  more  favorable  it  is  to  the  more  interactive 
(generally  the  videodisc)  treatment.  Generally,  costs  were  not  reported  in  these  studies, 
although  many  authors  acknowledged  cost  and  cost-effectiveness  to  be  significant  issues. 
The  abbreviation  "na"  is  used  in  the  Cost  Ratio  column  to  signify  that  the  data  of  interest 
were  not  reported  and  not  available. 

Cost  models  usually  cover  costs  in  four  categories:  research  and  development, 
initial  investment,  operating  and  support,  and  recovery  and  salvage.  All  cost  ratios  found 
and  reported  in  Appendix  A  are  based  on  initial  investment  (□),  operating  and  support  (OS) 
costs,  or  both  combined. 

An  assumption  made  in  aggregating  effect  sizes  is  that  each  effect  size  is  obtained 
independently  of  the  others  and  that  it  independently  estimates  the  true  effect  size. 
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However,  several  investigations  listed  in  Appendix  A  measured  more  than  one  instructional 
outcome  for  the  same  experimental  condition  and  for  the  same  experimental  subjects. 
These  outcomes  are  non-independent,  correlated  estimators  of  true  effect  size  and  should 
not  be  treated  as  if  they  were  independent.  Hedges  and  Olkin  (1985)  present  methods  of 
estimating  a  common  effect  size  from  a  vector  of  correlated  estimators.  Unfortunately,  the 
correlations  needed  to  apply  their  methods  are  rarely  reponed.  Further,  as  Hedges  and 
Olkin  point  out,  the  gain  in  efficiency  resulting  from  using  their  multivariate  methods 
seldom  justifies  the  required  effort.  An  alternative  approach  is  simply  to  average  non- 
independent  effect  sizes  across  a  single  experiment,  and  that  is  the  approach  used  here. 

In  calculating  these  averages,  it  was  necessary  to  decide  what  were  separate 
experimental  comparisons,  independent  and  worthy  of  separate  entry,  and  what  were  not. 
When  more  than  one  result  was  reported  and  more  than  one  effect  size  was  calculated  from 
the  same  investigation,  the  following  assumptions  guided  these  decisions: 

(1 )  If  the  effect  sizes  were  based  on  comparisons  using  different  students/subjects, 
it  was  assumed  they  constituted  different  experiments  and  the  effect  sizes  were 
not  averaged. 

(2)  If  the  effect  sizes  were  based  on  comparisons  involving  different  (more  than 
one)  experimental  groups  with  the  same  control  group,  it  was  assumed  they 
were  different  measures  of  different  experimental  conditions  and  the  effect 
sizes  were  not  averaged. 

(3)  If  the  effect  sizes  were  based  on  comparisons  involving  the  same  experimental 
treatment  (one  experimental  group)  but  different  (more  than  one)  control 
groups,  it  was  assumed  they  were  different  measures  of  the  same  experimental 
conditions  and  the  effect  sizes  were  averaged. 

(4)  Different  experiments  using  the  same  interactive  videodisc  instruction  materials 
were  not  averaged. 

In  Appendix  A,  a  superscript  "X"  in  the  ”ES  Comb  SD"  column  signifies  an 
average  effect  size  calculated  for  two  or  more  measures  reported  for  the  investigation-- 
"KX"  signifies  an  average  calculated  for  knowledge  outcomes,  "PX"  signifies  an  average 
calculated  for  performance  outcomes.  "AX”  signifies  an  overall  average  calculated  for 
knowledge,  performance,  retention,  and/or  duration  (time  to  complete  the  instruction) 
outcomes. 
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B.  EFFECTIVENESS 


1.  Principal  Outcomes 

Effect  sizes  for  the  four  principal  measures  of  merit— knowledge  outcomes, 
performance  outcomes,  retention  of  knowledge  or  performance,  and  time  to  complete  the 
instruction- were  averaged  across  all  28  studies  and  are  shown  in  Table  1.  The  effect  sizes 
reported  here  and  elsewhere  are  organized  into  three  columns: 

•  Column  one  shows  the  average  effect  size  for  all  available  studies  for  the  row 
category.  The  effect  sizes  entered  into  this  average  are  all  based  on  effect  sizes 
calculated  from  pooled  standard  deviations. 

•  Column  two  also  shows  average  effect  sizes  calculated  from  pooled  standard 
deviations.  However,  they  are  based  only  on  those  studies  that  presented 
sufficient  data  to  allow  both  pooled  and  control  group  standard  deviations  to  be 
used  in  calculating  effect  size. 

•  Column  three  shows  average  effect  sizes  calculated  using  control  group 
standard  deviations  only. 

All  the  effect  size  averages  in  Table  1  that  were  calculated  using  pooled  standard 
deviations  are  larger  than  those  calculated  using  control  group  standard  deviations.  This  is 
a  stable  finding  in  this  review.  It  indicates  that  control  group  standard  deviations  were 
generally  larger  than  experimental  group  standard  deviations  and  that  interactive  videodisc 
instruction  raised  average  achievement  more  equitably  across  all  students  than  did 
conventional  instruction.  The  increases  in  achievement  under  interactive  videodisc 
instruction  were  not  due  to  a  few  students  who  were  permitted  to  surge  ahead  in 
achievement  (and  increase  the  variance).  This  result  is  especially  important  for  training 
where  a  principal  concern  is  to  raise  as  many  students  as  possible  across  a  threshold— or 
thresholds-of  achievement  rather  than  simply  to  maximize  average  achievement,  which  is 
accomplished  most  efficiently  by  concentrating  instructional  resources  on  the  best  students. 

The  bottom  section  of  Table  1  shows  overall  average  effect  sizes  aggregated  across 
all  studies  and  all  four  measures  of  merit.  The  number  of  measures  that  contributed  to  the 
overall  effect  sizes  reported  in  the  table  is  less  than  the  sum  of  measures  contributing  to  the 
other  effect  sizes  since  some  within-study  measures  of  merit  for  a  single  experimental 
condition  were  averaged  together.  These  within-study  averages,  which  were  used  to 


calculate  the  overall  average  effect  sizes  reported  in  Table  1,  are  shown  in  Appendix  A  and 
signified  with  an  "X"  as  discussed  above. 

Table  i.  Average  Effect  Sizes  for  Knowledge  Outcomes,  Performance 
Outcomes,  Retention,  and  Instruction  Completion  Time 


ES  Based  on 
_ Pooled  SD _ 

ES  Based  on 
Pooled  SD 

When  Ctrl  SD 
_ Also  Available _ 

ES  Based 
_ onCtrlSQ _ 

Knowledge 

Outcomes 

Mean 

0.361 

0.375 

0.361 

SD 

(0.493) 

(0.506) 

(0.533) 

N 

(27) 

(23) 

(23) 

Perlorniaace 

Outcomes 

Mean 

0.349 

0.398 

0.334 

SD 

(0.556) 

(0.569) 

(0.447) 

N 

(20) 

(15) 

(15) 

Retention 

Mean 

0.547 

0.750 

0.650 

SD 

(0.511) 

(0.523) 

(0.339) 

N 

(3) 

(2) 

(2) 

Time  to  Complete 
instruction 

Mean 

1.178 

1.267 

1.185 

SD 

(0.382) 

(0.474) 

(0.570) 

N 

(8) 

(5) 

(5) 

Overall 

Mean 

0.502 

0.487 

0439 

SD 

(0.552) 

(0.563) 

(0.524) 

N 

(47) 

(38) 

(38) 
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Table  1  shows  47  knowledge,  performance,  retention,  and  time  to  complete 
measures  averaged  to  provide  an  overall  measure  of  merit  for  interactive  videodisc 
instruction  compared  to  conventional  instruction.  Of  these,  38  were  positive.  The  effect 
sizes  ranged  from  -1.14  to  2.18.  Their  average  is  0.50,  and  their  median  value  is  0.26  for 
effect  sizes  based  on  pooled  standard  deviations  and  0.28  for  effect  sizes  based  on  control 
group  standard  deviations.  Their  average  of  0.50  is  20-80  percent  larger  than  effect  sizes 
found  in  meta-analyses  of  computer-based  instruction-such  as  those  summarized  by 
Niemiec  and  Walberg  (1987). 

Although  effect  size  appears  to  be  an  excellent  metric  for  aggregating  the  results  of 
many  different  studies,  its  implications  for  the  practical  world  of  instruction  are  not 
immediately  evident.  A  mean  effect  size  of  0.50  from  a  sample  as  large  as  47  suggests  that 
through  the  use  of  interactive  videodisc  technology,  a  trainer  can  expect  to  increase  the 
achievement  of  students  by  about  one-half  a  standard  deviation  above  their  present  level  of 
achievement.  Or,  assuming  a  normal  or  Gaussian  distribution,  the  average  graduate,  the 
graduate  at  the  50th  percentile,  would  improve  to  the  level  of  graduates  currently  at  the  69th 
percentile  through  the  use  of  interactive  videodisc  technology.  Military  training  is  more 
concerned  with  numbers  of  students  who  reach  defined  thresholds  of  knowledge  or 
performance  per  constant  expenditure,  than  with  increases  in  their  achievement.  In  less 
normative  terms  then,  and  assuming  a  linear  rate  of  learning,  an  effect  size  of  0.50 
suggests,  roughly,  an  improvement  of  19  percent  in  number  of  graduates  through  the 
application  of  interactive  videodisc  instruction. 

Perfoimance  and  knowledge  outcomes  are  reported  separately  in  Table  1.  These 
results  show  that  die  average  effect  sizes  for  these  two  achievement  measures  are  similar, 
and  suggest  that  there  is  little  reason  to  expect  interactive  videodisc  instruction  to  be  any 
more  effective  for  performance  outcomes  than  for  knowledge  outcomes,  although  in  both 
cases  it  is  more  effective  than  conventional  instruction.  The  average  effect  sizes  for 
performance  outcomes  range  from  0.334  for  15  studies  based  on  control  group  standard 
deviations  to  0.398  for  the  same  studies  based  on  pooled  standard  deviations.  The  average 
effect  sizes  for  knowledge  outcomes  range  from  0.361  for  23  studies  based  on  control 
group  standard  deviations  to  0.375  for  the  same  studies  based  on  pooled  standard 
deviations.  These  results  are  generally  the  same  for  both  knowledge  and  performance 
outcomes.  They  suggest  an  improvement  of  50th  percentile  students  to  about  64th 
percentile  achievement  or  an  increase  of  about  14  percent  in  the  number  of  students 
reaching  criterion  achievement  levels. 
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Average  effect  sizes  for  time  to  complete  instruction--or  to  reach  criterion  level(s)  of 
achievement— are  also  shown  in  Table  1.  These  averages  range  from  1.178  for  8  effect 
sizes  calculated  across  all  available  studies  to  1 .267  for  5  effect-  sizes  based  on  pooled 
standard  deviations.  These  results  suggest  an  improvement  of  50th  percentile  students  to 
about  the  88th  percentile  achievement,  or  an  increase  of  38  percent  in  the  number  of 
students  reaching  criterion  achievement  levels  if  time  were  held  constant.  Student  time 
savings  of  about  30  percent  have  been  noted  in  analyses  of  computer  based  instruction 
starting  with  a  widely  noted  Orlansky  and  String  (1977)  study  and  continuing  on  into  the 
present.  The  average  percent  time  saved  across  the  8  studies  in  this  review  that  reported 
time  savings  was  31  percent.  Eight  studies  do  not  provide  a  conclusive  base,  but  these 
results  suggest  that  instruction  time  savings  for  interactive  videodisc  instruction  are  about 
the  same  as  those  for  computer-based  instruction. 

One  might  well  ask  to  what  degree  these  time  savings  were  due  to  self-pacing,  to 
what  degree  they  were  due  to  self-pacing  uniquely  made  possible  by  interactive  videodisc 
instruction,  and  to  what  degree  self-pacing  was  encouraged  in  the  conventional  instruction 
with  which  the  videodisc  instruction  in  these  studies  was  compared.  These  questions  were 
not  well  addressed  by  the  studies  summarized  in  this  review.  Generally  the  treatments 
were  designed  to  allow  the  same  flexibilities  for  completing  conventional  instruction  as  for 
interactive  videodisc  instruction,  and  results  from  these  studies  must  stand  until  more 
focused  research  is  performed  and  reported. 

2 .  Instructional  Settings 

Overall  average  effect  sizes  for  all  four  measures  of  merit  were  broken  out  of  Table 
1  and  calculated  separately  for  military,  industrial,  and  higher  education  settings.  They  are 
shown  in  Table  2.  The  average  effect  size  for  military  applications  ranges  from  0.385, 
based  on  20  studies  calculated  using  control  group  standard  deviations  to  0.416  based  on 
the  same  experimental  comparisons  but  calculated  using  pooled  standard  deviations.  These 
results  suggest  an  improvement  of  50th  percentile  students  to  about  65th  percentile 
achievement  or  an  increase  of  15  percent  in  the  number  of  students  reaching  criterion 
achievement,  assuming  linear  learning.  With  over  250,000  students  involved  in 
residential,  formal  instruction  programs  each  day,  these  findings  promise  significant 
improvements  for  Defense  training  provided  that  the  operations  of  schools  are  changed  to 
take  advantage  of  these  improvements. 


The  average  effect  sizes  for  interactive  videodisc  instruction  used  in  higher 
education  are  considerably  greater  than  those  for  military  training.  In  Table  2,  they  range 
from  0.660  for  12  studies  based  on  control  group  standard  deviations  to  0.725  based  on 
the  same  experimental  comparisons  but  calculated  using  pooled  standard  deviations.  These 

Table  2.  Average  Effect  Sizes  for  Military  Training,  Industrial  Training,  and 

Higher  Education 


ES  Based  on 
Pooled  SD 

ES  Based  on  When  Ctrl  SD  ES  Based 

Pooled  SD _ Also  Available _ QQ.Ctil.SD_ 


Military  Training 

Mean  0.391  0.416 

SD  (0.543)  (0.543) 

N  (24)  (20) 


0.385 

(0.451) 

(20) 


Industrial  Iiainina 

Mean  0.510  0.249 

SD  (0.744)  (0.799) 

N  (9)  (6) 


0.174 

(0.838) 

(6) 


Higher  Education 

Mean  0.689  0.725 

SD  (0.395)  (0.408) 

N  (14)  (12) 


0  .660 
(0.393) 
02) 


All  Settings  Combined 

Mean  0.502  0.487 

SD  (0.552)  (0.563) 

N  (47)  (38) 


0.439 

(0.524) 

(38) 


results  suggest  an  improvement  of  50th  percentile  students  to  about  75th  percentile 
achievement  or  an  increase  of  25  percent  in  the  number  of  students  reaching  criterion 
achievement.  Since  the  emphasis  in  training  is  properly  on  reducing  costs  to  reach 
threshold  levels  of  achievement,  this  emphasis  may  actually  be  observed  in  practice.  The 
lower  effect  sizes  found  for  military  training  may  be  due  to  this  observance-students  in 
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training  who  reach  threshold  criteria  are  sent  on  to  new  assignments,  whereas  students  in 
education  are  encouraged  to  continue  improving.  If  this  argument  holds,  we  should  expect 
to  find  effect  sizes  for  industrial  training  that  are  closer  to  those  of  military  training  than  to 
those  of  higher  education. 

Unfortunately,  the  findings  for  industrial  training  are  less  stable  than  those  for 
either  military  training  or  higher  education.  In  Table  2,  they  range  from  0. 174  for  6  studies 
based  on  control  group  standard  deviations  to  0.510  for  9  studies  based  on  pooled  standard 
deviations.  The  improvement  of  50th  percentile  students  suggested  by  these  results  ranges 
from  57th  to  69th  percentile  achievement  or  an  increase  in  the  number  of  students  reaching 
criterion  achievement  levels  of  7-19  percent.  The  three  additional  studies  that  are  included 
in  the  effect  size  average  shown  in  Column  1,  are  all  based  on  time  to  complete  training. 
The  remaining  6  studies,  upon  which  the  Column  2  and  Column  3  effect  size  averages  are 
based,  are  either  knowledge  or  performance  outcomes  measured  when  the  students 
completed  the  instruction. 

The  pattern  of  knowledge  and  performance  measures  across  all  three  instructional 
settings  is  therefore  of  interest  and  is  reported  in  Table  3.  As  this  table  suggests,  the  effect 
sizes  from  industrial  training  appear  much  closer  to  those  of  military  training  than  to  those 
of  higher  education  if  we  limit  consideration,  as  Table  3  does,  to  measures  of  performance 
and  knowledge  taken  at  the  end  of  instruction. 

3 .  Instructional  Approaches 

Another  exploration  concerned  the  influence  of  instructional  approach— how  the 
videodisc  system  was  used— on  instructional  outcomes.  The  studies  were  categorized  into 
three  groups  based  on  their  instructional  approach:  studies  in  which  the  instructional 
approach  was  directive  and  tutorial;  studies  in  which  the  instructional  approach  was  based 
on  freely  explored  simulations;  and  studies  in  which  tutorial  and  simulation  approaches 
were  combined.  These  studies  were  crossed  with  those  in  which  the  instructional  outcome 
assessed  was  knowledge  or  performance,  which  yielded  the  four  (tutorial,  combined, 
simulation,  all  three)  by  three  (knowledge,  performance,  combined  knowledge  and 
performance)  set  of  results  shown  in  Table  4.  Not  surprisingly,  there  are  too  few  studies 
in  some  of  these  categories.  For  instance,  there  was  only  one  study  in  which  a  tutorial 
approach  was  used  to  produce  a  performance  outcome.  It  should  be  emphasized  that  the 
categorization  of  instructional  approaches  in  this  review  was  generally  based  on  evaluation 
documentation  and  not  on  direct  experience  with  the  instructional  materials  themselves. 
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Table  3.  Average  Effect  Sizes  for  Combined  Knowledge  and  Performance 
Outcomes  In  Military  Training,  Industrial  Training,  and  Higher  Education 


ES  Based  on 
Pooled  SD 

ES  Based  on 
Pooled  SD 

When  Ctrl  SD 

Also  Available 

ES  Based 
on  Ctrl  SD 

Military  Training 

Mean 

0.353 

0.380 

0.357 

SD 

(0.541) 

(0.533) 

(0.446) 

N 

(23) 

(19) 

(19) 

Mean 

0.249 

0.249 

0.174 

SD 

(0.744) 

(0.799) 

(0.838) 

N 

(6) 

(6) 

(6) 

Higher  Education 

Mean 

0.582 

0.601 

0.554 

SD 

(0.354) 

(0.373) 

(0.355) 

N 

(13) 

(11) 

(11) 

All.  Settings  Combined 

Mean 

0.409 

0.464 

0.388 

SD 

(0.536) 

(0.541) 

(0.507) 

N 

(42) 

(36) 

(36) 

There  appear  to  be  two  main  results  from  this  analysis.  First,  there  seems  to  be 
about  the  same  amount  of  improvement  in  knowledge  and  performance  outcomes  for  dll 
approaches.  There  is  some  minor  evidence  here  suggesting  that  simulation  either  alone  or 
in  combination  with  a  tutorial  yields  stronger  performance  outcomes  than  knowledge 
outcomes.  This  Finding  seems  intuitively  reasonable,  but  it  is  too  weak  to  do  anything  but 
suggest  a  line  of  follow-on  investigation. 

Second,  there  is  an  obvious  increase  in  effectiveness  across  these  studies  from 
simulation-only  approaches,  which  were  least  effective,  through  approaches  combining 
simulation  and  tutorials,  to  tutorial  approaches,  which  were  most  effective.  Because  of 


Table  4.  Average  Effect  Sizes  for  Tutorial  Approaches  and 
Instructional  Outcomes 


ES  Based  on 

Instruc- 

Pooled  SD 

tlonal 

ES  Based  on 

When  Ctrl  SD 

ES  Based 

Approach 

Measure  Pooled  SO 

Also  Available 

on  Ctrl  SD 

Knowledge 

Tutorial 

Mean 

0.703 

0.703 

0.660 

Outcomes 

SD 

(0.249) 

(0.249) 

(0.234) 

N 

(6) 

(6) 

(6) 

Combined 

Mean 

0.398 

0.360 

0.385 

Tutorial  & 

SD 

(0.420) 

(0.452) 

(0.472) 

Simulation 

N 

(12) 

(10) 

(10) 

Simulation 

Mean 

0.104 

0.102 

0.033 

SD 

(0.536) 

(0.604) 

(0.629) 

N 

(9) 

(7) 

(7) 

All 

Mean 

0.361 

0.375 

0.361 

Approaches 

SD 

(0493) 

(0.506) 

(0.533) 

N 

(27) 

(23) 

(23) 

Performance 

Tutorial 

Mean 

0.805 

0.805 

0.805 

Outcomes 

SD 

(na) 

(na) 

(na) 

N 

(i) 

(D 

(1) 

Combined 

Mean 

0.423 

0.447 

0.341 

Tutorial  & 

SD 

(0.692) 

(0.746) 

(0.554) 

Simulation 

N 

(11) 

(8) 

(8) 

Simulation 

Mean 

0.191 

0.265 

0.250 

SD 

(0.299) 

(0.261) 

(0.284) 

N 

(8) 

(6) 

(6) 

All 

Mean 

0.349 

0.398 

0.334 

Approaches 

SD 

(0.556) 

(0.569) 

(0.447) 

N 

(20) 

(15) 

(15) 

Combined _ 

Tutorial 

Mean 

0.718 

0.718 

0.678 

Knowledge 

SD 

(0.230) 

(0.230) 

(0.219) 

&  Performance 

N 

(7) 

(7) 

(7) 

Outcomes 

Combined 

Mean 

0.410 

0.398 

0.365 

Tutorial  & 

SD 

(0.553) 

(0.583) 

(0.495) 

Simulation 

N 

(23) 

(18) 

(18) 

Simulation 

Mean 

0.145 

0.177 

0.133 

SD 

(0.430) 

(0.467) 

(0.494) 

N 

(17) 

(13) 

(13) 

Ail 

Mean 

0.409 

0.464 

0.388 

Approaches 

SD 

(0.536) 

(0.541) 

(0.507) 

N 

(47) 

(38) 

(38) 

cost  considerations,  there  is  little  here  that  argues  against  use  of  simulation.  The  cost 
arguments  for  simulation  are  very  strong  when  a  few  thousand  dollars  of  software  and 
hardware  can  substitute  for  actual  equipment  or  field  experiences  that  may  cost  tens  or 
hundreds  of  thousands  of  dollars.  Results  from  this  review  show  that  such  substitutions 
are  both  feasible  and  desirable.  Cost  arguments  for  tutorial  approaches  are  seldom  this 
strong  since  tutorials  can  usually  be  presented  by  relatively  inexpensive  means  such  as 
textbooks,  programmed  textbooks,  or  teaching  assistants  as  well  as  by  videodisc  hardware. 
Moreover,  simulation  only  approaches  showed  average  effect  sizes  ranging  from  0.033 
(for  knowledge  outcomes)  to  0.265  (for  performance  outcomes).  They  were,  overall, 
more  effective  than  conventional  approaches.  These  results,  combined  with  the  favorable 
cost  ratios  that  are  generally  found  in  simulation  approaches,  suggest  that  videodisc 
instruction  using  simulation  alone  may  turn  out  to  be  the  most  cost-effective  alternative 
even  though  it  may  be  the  least  effective  of  the  three  approaches  examined  here. 

The  greater  effect  sizes  observed  for  tutorial  approaches  over  combined  tutorial  and 
simulation  approaches  are  puzzling  for  at  least  two  reasons.  First,  it  seems  intuitively 
reasonable  that  including  simulation  with  a  tutorial  should  add  both  value  to  the  tutorial  and 
effectiveness  to  the  instruction.  Just  the  opposite  is  suggested  by  these  data.  Second,  the 
programmed  instruction  approach  used  in  tutorial  instruction  is  typically  reported  by 
reviews  of  paper-based  (programmed  textbook)  approaches  to  produce  substantially  lower 
effect  sizes  than  those  found  here.  For  instance,  Kulik,  Cohen,  and  Ebeling  (1980)  found 
that  57  studies  of  programmed  instruction  used  in  higher  education  to  present  a  variety  of 
subject  areas  improved  performance  by  an  average  0.24  standard  deviations  over 
conventional  instruction,  which  is  considerably  less  than  the  effect  sizes  for  tutorial 
approaches  reported  in  this  review.  The  effectiveness  of  tutorial  approaches  used  in 
interactive  videodisc  instruction  evidently  deserves  further  investigation. 

The  finding  that  directive,  tutorial  approaches  yield  greater  instructional  results  than 
simulation  approaches  is  supported  by  Gay  (1986)  who  used  a  2-by-2  experimental  design 
and  interactive  videodtsc  instruction  to  investigate  the  effects  of  learner  and  program  control 
on  the  achievement  of  students  who  had  high  and  low  prior  knowledge  of  course  content. 
Students  in  the  learner  control  (non-tutorial)  condition  with  low  prior  knowledge  scored 
significantly  lower  in  a  post-test  of  course  content  than  did  students  in  the  other  three 
treatment  groups.  Gay’s  results  along  with  those  of  this  review  suggest  that  directive, 
tutorial  approaches  may  be  best  used  when  standardized  results  are  to  be  obtained  from 
students  with  a  variety  of  prior  knowledge,  which  is  the  typical  case  in  military  training. 
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On  the  other  hand,  high  prior  knowledge  students  in  Gay's  study  who  were  in  the 
learner  control  condition  took  significantly  less  time  to  complete  the  course  and  the  post-test 
than  did  students  in  any  of  the  other  three  treatment  groups.  This  result  may  explain  some 
of  the  mixed  findings  for  directive,  tutorial  approaches  found  by  this  review,  and  it 
suggests  that  important  efficiencies  in  training  may  result  if  the  training  approach  can  be 
adjusted  for  prior  knowledge  of  course  content. 

4 .  Level  of  Interactivity 

Average  effect  sizes  from  five  within-videodisc  comparisons  (i.e.,  those  that 
compare  high  levels  of  videodisc  interactivity  with  lower  ones)  are  shown  in  Table  5. 
These  range  from  0.456  for  effect  sizes  based  on  control  group  standard  deviations  to 
0.397  for  effect  sizes  based  on  pooled  standard  deviations.  The  individual  effect  sizes  that 
contribute  to  the  averages  reported  in  Table  5  range  from  a  single  negative  effect  size  of  - 
-0.160  to  1.425.  Since  there  are  no  absolute  levels  of  interactivity— one  study's  high  level 
of  interactivity  might  be  another  study's  low  level  of  intcractivity--it  is  difficult  to  interpret 
the  magnitudes  of  these  effect  sizes  directly.  However,  these  studies  suggest  that  the 
greater  the  level  of  interactivity,  the  greater  the  instructional  achievement  provided  by  the 
videodisc  instruction. 


Table  5.  Average  Effect  Sizes  for  High  Compared  with  Low  Levels  of 

videodisc  Interactivity 


Measures 

ES  Based  on 
Pooled  SO 

ES  Based  on 
Pooled  SD 

When  Ctrl  SD 

Also  Available 

ES  Based 
on  Ctrl  SD 

Mean 

0.397 

0.397 

0.456 

(0.533) 

(0.533) 

(0  631) 

N 

(5) 

(5) 

(5) 

Notably,  the  Table  5  effect  sizes  based  on  pooled  standard  deviations  are  less  than 
the  effect  sizes  based  on  control  group  standard  deviations.  This  result  is  contrary  to 
almost  all  the  previous  effect  size  results.  It  means  that  the  standard  deviations  across  the 
five  control  groups  were  smaller  than  those  of  the  experimental  groups,  and  it  suggests  that 
when  different  levels  of  interactivity  are  compared  within  interactive  videodisc  approaches, 
greater  interactivity  is  associated  with  greater  instructional  outcome  variability.  A  result 
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based  on  only  five  studies  cannot  be  viewed  as  conclusive,  but  it  does  suggest  a  variable  to 
be  considered  in  future  research. 

Four  studies  of  Level  II  videodisc  instruction  appeared  in  our  search  for  empirical 
evaluations.  These  studies  are  summarized  in  Appendix  B.  As  the  Appendix  shows, 
results  from  these  evaluations  are  generally  less  positive  than  those  reported  for  Levei  III 
applications.  For  that  matter,  five  of  the  seven  results  that  were  statistically  significant 
were  negative,  showing  the  videodisc  application  to  be  less  effective  than  conventional 
instruction.  Of  course  the  lesser  effectiveness  of  the  videodisc  instruction  may  be  balanced 
out  by  lower  cost.  Unfortunately,  none  of  these  studies  reported  cost  ratios,  and  the  cost 
effectiveness  of  these  applications  remains  unknown. 

Table  8  summarizes  the  average  effect  sizes  for  Level  II  evaluations  for  knowledge, 
performance,  and  time  to  completion  measures.  Performance  outcomes  appear  much  lower 
than  knowledge  outcomes  among  these  effect  sizes,  ranging  ffom-1.752  and-1.565  for  six 
performance  outcomes  to  0.385  and  0.350  for  two  knowledge  outcomes.  There  are  too 
few  studies  here  to  view  this  as  a  reliable  result.  Superiority  of  knowledge  outcomes  to 
performance  outcomes  from  Level  II  videodisc  presentations  must  remain  for  the  present  an 
interesting  possibility. 

The  overall  averages  in  Table  8  range  from  -1.031  for  an  average  of  10  effect  sizes 
based  on  control  group  standard  deviations  to*0.7G2  for  13  effect  sizes  based  on  pooled 
standard  deviations.  They  are  large  enough  to  be  considered  seriously.  Across  the  studies 
reported  here,  videodisc  instruction  based  on  Level  II  technology  appears  much  less 
effective  than  conventional  instruction. 

The  Table  6  average  effect  sizes  based  on  pooled  standard  deviations  are  smaller  in 
absolute  magnitude  than  those  based  on  control  group  standard  deviations.  This  result 
suggests  that  there  was  generally  less  variation  in  the  conventional  instruction  groups  than 
in  the  Level  II  videodisc  instruction  groups.  Since  the  videodisc  instruction  groups 
generally  achieved  lower  scores  on  the  outcome  measures  considered  in  the  Level  II 
studies,  it  may  be  that  some  students  simply  did  not  adapt  to  the  videodisc  presentations 
and  found  themselves  unable  to  learn  effectively  from  videodisc  approaches-whereas  other 
students  progressed  just  as  well  using  videodiscs  as  they  would  in  conventional  • 

instruction.  This  possibility  could  be  investigated  by  examining  the  raw  data  from  these 
studies. 
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Table  6.  Average  Effect  Sizes  for  Knowledge  Outcomes,  Performance 
Outcomes,  and  Instruction  Completion  Times  for  Instruction 
Using  Level  II  Videodiscs 


ES  Based  on 
Pooled  SD 

ES  Based  on 
Pooled  SD 

When  Ctrl  SD 

Also  Available 

ES  Based 
on  Ctrl  SD 

Knowledge 

Outcomes 

Mean 

0.115 

0.350 

0.385 

SD 

(0.276) 

(0.057) 

(0.120) 

N 

(4) 

(2) 

(2) 

Performance 

Outcomes 

Mean 

-1.565 

-1.565 

-1.752 

SD 

(1.331) 

(1.331) 

(1  526) 

N 

(6) 

(6) 

(6) 

Time  to  Complete 

Instruction 

Mean 

-0.067 

-0.245 

-0.285 

3D 

(0.330) 

(0.163) 

(0.163) 

N 

(3) 

(2) 

(2) 

Overall 

Mean 

-0.702 

-0.918 

-1.031 

SD 

(-1.213) 

(1.314) 

(1.488) 

N 

(13) 

(10) 

(10) 

The  lower  levels  of  interactivity  provided  by  Level  II  videodisc  systems  may 
account  for  the  lower  levels  of  effectiveness  found  in  these  studies.  Although  this  is  not 
the  only  explanation,  interactivity  is  one  of  the  main  differences  between  the  studies 
summarized  in  Appendix  B  and  those  summarized  in  Appendix  A.  Notably,  six  of  these 
results  came  from  the  Holmgren  et  al.  (1979)  study  in  which  Army  Training  Extension 
courses  designed  to  be  presented  using  audio  tape  and  film  slides  were  directly  recorded  on 
videodisc.  The  findings  of  this  study  suggest  that  tape-slide  materials  presented  on  a  tape- 
slide  medium  will  produce  training  results  that  are  superior  to  those  obtained  when  the 
same  materials  are  presented  using  another  medium. 
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These  findings  further  suggest  thai.  the  less  the  unique  capabilities  of  interactive 
videodiscs  are  used,  the  less  the  advantage  gained  from  using  the  medium.  This  possibility 
is  supported  by  comparison  of  the  Wankel  (1984)  with  the  Stevens  (1984)  results.  The 
same  basic  videodisc  material  ("Puzzle  of  the  Tacoma  Bridge  Collapse")  was  used  in  both 
cases,  and  in  both  cases  the  same  test  of  physics  knowledge  was  used.  When  a  Level  II 
approach  was  used,  the  effect  size  was-0.17;  when  a  Level  III  approach  was  used,  the 
effect  size  was  0.29.  This  possibility  is  also  supported  by  Verano  (1987),  who  presented 
three  levels  of  interactivity  (linear,  segmented,  and  interactive)  to  students.  The  higher  the 
level  of  interactivity  he  used,  the  greater  the  students'  achievement.  Instructional  evaluation 
generally  requires  a  sizable  body  of  research  to  achieve  a  conclusive  result,  but  the 
combination  of  the  Holmgren,  Wankel,  Stevens,  and  Verano  results  as  well  as  the  contrast 
between  Level  II  and  Level  HI  effect  sizes  make  a  strong  case  for  using  the  full  capabilities 
of  Level  EU  videodisc  media  in  instructional  applications. 

Extension  of  this  conclusion  to  Level  IV  videodisc  materials  is  probably  not 
warranted.  The  step  from  Level  HI  to  Level  IV,  under  the  definitions  used  for  this  review, 
only  marks  a  change  in  the  way  instructional  materials  are  stored  and  not  any  improvement 
in  the  way  they  are  presented.  For  that  matter,  the  proprietary  nature  of  all  current 
approaches  to  encoding  and  decoding  digital  information  on  and  off  videodiscs  coupled 
with  the  lack  of  additional  instructional  capabilities  makes  Level  IV  videodisc  technology 
incompatible  with  current  Defense  initiatives  to  achieve  courseware  portability  (Fletcher, 
1988). 

5 .  Evaluation  Issues 

The  above  findings  suggest  favorable  results  for  Level  III  interactive  videodisc 
instruction  relative  to  conventional  instruction.  They  arise  directly  from  available 
evaluation  data  on  the  effects  of  interactive  videodisc  applications  in  training  and  education. 
However,  there  are  two  issues  to  note  concerning  the  studies  from  which  these  results  are 
drawn: 

(1)  Most  of  the  evaluations  reported  here  were  performed  by  developers  who  also 
produced  the  interactive  videodisc  instruction  being  evaluated.  Few  of  the 
evaluations  were  performed  by  third  parties.  There  are  strengths  and 
weaknesses  in  this  approach.  Developers  are  rarely  indifferent  to  the  success 
of  their  products  and  may,  intentionally  or  not,  bias  the  results  of  their 
evaluation.  On  the  other  hand,  there  are  arguments  that  support  initial 
evaluation  of  instructional  materials  by  its  developers  and  not  by  a  third  party 
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evaluator.  For  example,  there  are  standards  for  the  performance  and 
documentation  of  evaluation  studies.  Observance  of  these  standards  can  be 
assessed  by  others--as  they  were  in  the  search  for  evaluation  studies 
summarized  here— and  bias  in  studies  performed  by  developers  and  selected  for 
this  review  should  be  minimal.  Also,  developers  have  a  stake  in  an  honest 
assessment  of  their  products.  The  better  the  information  they  get,  the  better 
their  practice  and  products  can  become.  Finally,  developers  may  be  in  the  best 
position  to  evaluate  their  products.  They  may  understand  better  than  anyone- 
including  potential  users— the  strengths  and  limitations  of  what  they  have 
developed,  and  they  are  better  prepared  than  anyone  to  assess  their  products. 

(2)  One  difficulty  for  any  evaluation  of  an  emerging  technology  is  that  there  is 
nothing  else  like  it.  Each  new  technology  has  its  own  strengths  and 
limitations.  If  the  evaluation  is  held  to  strict  instructional  and  experimental 
controls  based  on  the  older  technology,  the  newer  technology  will  be  at  a 
disadvantage.  This  problem  can  be  ameliorated  somewhat  by  focusing  on 
instructional  outcomes  and  not  on  what  is  done  to  accomplish  these  outcomes. 
Even  in  the  best  of  situations,  however,  new  approaches  are  unlikely  to  be 
used  well  since  not  enough  is  understood  about  how  best  to  employ  them. 
Despite  all  the  evaluations  listed,  we  may  have  yet  to  see  an  approach  to 
instruction  that  uses  interactive  videodisc  technology  to  best  advantage. 

In  sum,  it  can  be  argued  either  that  the  view  of  interactive  videodisc  applications 
seen  here  is  overly  optimistic  or  overly  pessimistic.  At  present  it  seems  best  to  rely  on  the 
data  we  have  and  to  conclude  that  interactive  videodisc  instruction  offers  genuine  and 
demonstrable  gains  in  instructional  effectiveness  over  the  conventional  approaches  we  use 
now,  even  though  we  understand  little  about  how  these  gains  come  about. 

C.  COST 

Effectiveness  by  itself  is  only  half  the  story.  Military  trainers,  along  with  their 
civilian  counterparts,  are  more  frequently  asking  what  their  investment  in  instructional 
technology  has  bought  in  increased  instructional  effectiveness.  Cost  ratios  (calculated  as 
the  ratio  of  experimental  treatment  costs  over  control  treatment  costs)  were  used  to  assess 
costs.  Ratios  in  this  investigation  that  are  less  than  1,0  indicate  lower  costs  for  interactive 
videodisc  instruction  than  for  conventional  instruction. 
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Thirteen  cost  ratios  are  reported  in  Appendix  A.  All  of  these  ratios  are  less  than 
1.0.  Most  came  from  studies  concerned  with  training  applications.  Six  of  the  studies 
involving  military  training  reported  cost  ratios  (Wilkinson,  1983;  Cicchinelli  et  al.,  1984; 
Pieper  et  al.,  1984;  Vernon,  1984;  Green  et  a!.,  1986;  and  Winkler  and  Polich,  in  press); 
one  of  the  studies  (Aukerman,  1986)  involving  industrial  training  did;  and  one  of  the 
studies  (Davis,  1985)  involving  higher  education  did.  Five  of  the  cost  ratios  reported  are 
for  initial  investment  (II),  four  are  for  operating  and  support  costs  (OS),  and  four  for  some 
combination  of  initial  investment  with  operating  and  support  costs.  These  ratios  are  all 
calculated  as  the  costs  of  interactive  videodisc  instruction  over  the  costs  of  conventional 
instruction.  Since  there  are  too  few  ratios  here  to  provide  reliable  averages,  the  cost 
findings  from  these  eight  studies  and  1 3  rados  are  briefly  summarized  below. 

Wilkinson  (1983)  compared  the  costs  of  using  actual  equipment  with  simulations  of 
the  equipment  presented  by  interactive  videodisc  in  electronics  maintenance  training. 
Hardware  costs  of  the  videodisc  system  were  $7,180.  Total  software  production  costs 
were  $18,700,  but  costs  to  duplicate  the  software  were  negligible  so  that  the  marginal  costs 
for  the  software  declined  with  each  videodisc  station  equipped  with  it.  These  costs  should 
be  compared  to  the  standard  cost  of  $51,000  for  each  workstation  involving  actual 
equipment.  The  per  unit  costs  of  actual  equipment  workstations  would  decrease  only 
slightly  with  increases  in  quantities  purchased.  The  training  course  personnel 
recommended  that  a  minimum  of  eight  workstations  be  provided  for  student  use,  and  the 
cost  ratio  of  0.19  reported  in  Appendix  A  for  initial  investment  was  based  on  the 
assumption  that  eight  workstations  would  be  purchased. 

Cicchinelli  et  al.  (1984)  also  compared  the  costs  of  using  actual  equipment  versus 
videodisc  simulations  of  the  equipment  in  electronics  maintenance  training.  Their  findings 
indicated  that  the  videodisc-based  training  system  would  cost  slightly  more  than  the  training 
system  using  actual  equipment  for  the  first  copy  but  that  costs  for  subsequent  copies  would 
drop  off  quickly.  The  break-even  point  occurred  at  about  3  copies.  Cicchinelli  et  al. 
included  courseware  preparation  costs  with  costs  for  the  videodisc  system,  but  did  not 
include  courseware  costs  with  costs  for  the  actual  equipment  training  system  since 
courseware  costs  were  unavailable.  Consequently,  the  costs  for  interactive  videodisc 
instruction  were  somewhat  inflated  relative  to  the  costs  for  instruction  using  actual 
equipment. 

Cicchinelli  ct  al.  considered  two  "scenarios"  in  comparing  the  costs  of  simulations 
using  an  interactive  videodisc  system  in  place  of  actual  equipment.  In  one  scenario  they 
assumed  th;>t  10  actual  equipment  trainers  were  already  in  place  and  that  an  additional  10 
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were  needed.  The  additional  trainers  could  be  videodisc  systems  or  actual  equipment  The 
cost  ratio  of  videodisc  systems  to  older  (and  less  expensive)  actual  equipment  was  0.94  for 
the  additional  trainers.  The  ratio  was  0.84  for  videodisc  systems  over  newer  equipment 
actually  being  installed  in  operational  settings.  A  second  scenario  considered  the  costs  of 
modifying  10  videodisc  systems  to  reflect  engineering  changes  in  the  operational  equipment 
versus  modifying  the  actual  equipment  used  for  training.  Basically  the  comparison  was 
between  software  changes  in  the  videodisc  instruction  (15  new  lessons,  new  technical 
documentation,  and  videodisc  remastering)  versus  hardware  changes  in  the  actual 
equipment.  The  ratio  of  modification  costs  for  videodisc  instruction  over  actual  equipment 
instruction  in  this  scenario  was  0.44. 

Pieper  et  al.  (1984)  presented  another  study  of  simulation  based  on  interactive 
videodisc  versus  actual  equipment  in  electronic  maintenance  training.  The  ratio  of  single 
station  initial  investment  costs  for  hardware  was  0.02  for  a  videodisc-based  approach  over 
an  actual  equipment  approach.  The  similar  ratio  for  continuing  hardware  costs  was  also 
about  0.02.  The  overall  ratio  for  initial  investment  costs  including  hardware,  software, 
documentation,  and  administration  was  0.27  for  a  videodisc-based  approach  over  an  actual 
equipment  approach.  The  similar  overall  ratio  for  continuing  costs  was  0.03. 

Vemon  (1984)  compared  the  effectiveness  of  simulation  using  interactive  videodisc 
instruction  with  actual  equipment  to  train  soldiers  to  operate  data  communications 
terminals.  The  ratio  of  initial  investment  costs  including  hardware,  software,  and  all 
courseware  for  interactive  videodisc  training  over  actual  equipment  training  was  0.15.  The 
ratio  of  continuing  costs  including  the  amortized  initial  investment  for  hardware,  software, 
and  courseware  for  interactive  videodisc  training  over  actual  equipment  training  was  0.13. 
Considering  maintenance  costs  alone,  without  the  amortizing  initial  investment  costs,  the 
cost  ratio  for  interactive  videodisc  training  over  actual  equipment  training  was  0.01. 

Green  et  al.  (1986)  discussed  the  15-year  life  cycle  costs  for  the  approaches  studied 
by  Williams  and  Harold  (1985),  who  compared  classroom  instruction  alone  with  classroom 
instruction  supplemented  by  interactive  videodisc  materials.  Classroom  instruction  for  both 
groups  included  use  of  the  actual  equipment,  a  sonar  set  costing  $34  million  per  copy,  on 
which  students  were  learning  to  perform  maintenance.  The  ratio  of  costs  for  operating  the 
course  over  a  15-year  period,  including  the  initial  costs  of  hardware,  software,  and 
courseware,  was  0.05  for  the  substitution  of  about  17  hours  of  interactive  videodisc 
instruction  over  instruction  using  actual  equipment  for  that  segment  of  the  course. 


Winkler  and  Polich  (in  press)  compared  instruction  combining  interactive  videodisc 
and  actual  equipment  with  instruction  using  actual  equipment  only.  Costs  for  actual 
equipment  hardware  were  $138,000  per  copy  compared  with  $5,500  per  copy  for  the 
interactive  videodisc  hardware.  The  ratio  for  the  overall  combination  of  initial  investment 
and  operating  and  support  costs  for  interactive  videodisc  instruction  over  the  costs  for 
instruction  using  actual  equipment  was  0.17. 

Aukerman  (1986)  considered  only  student  and  instructor  time  costs.  She  found 
costs  of  about  $28.61  per  learner  for  about  2.6  hours  of  instruction  using  interactive 
videodisc  versus  about  $46.49  per  learner  using  lectures  and  discussion  yielding  a  cost 
ratio  of  0.61  for  interactive  videodisc  over  the  more  conventional  approach.  Most  of  the 
cost  differences  were  due  to  the  substantially  reduced  amount  of  instructor  time  required  for 
the  interactive  videodisc  instruction— about  8.5  minutes  versus  66  minutes  for 
lecture/discussion. 

Davis  (1985)  compared  wet-laboratory  facilities  with  videodisc-based  facilities  for 
instruction  in  respiration  for  biology.  She  did  not  include  instruction  preparation  costs  in 
her  analysis  for  either  of  these  approaches,  which  is  not  unreasonable  since  these  costs  may 
be  nearly  equal  or  they  may  be  trivial  compared  to  life  cycle  support  costs  of  both 
instructional  approaches,  but  she  did  include  space  preparation,  personnel,  supplies,  and 
equipment  costs.  Assuming  instruction  involving  three  different  organisms  kept  at  ttu  ee 
different  temperatures  and  a  need  to  renovate  space  for  both  instructional  approaches,  her 
cost  data  indicate  a  ratio  of  0.80  for  initial  investment  and  0.02  for  operating  and  support. 

Other  studies,  not  summarized  in  Appendix  A,  reported  useful  analyses  comparing 
the  costs  of  interactive  videodisc  instruction  with  conventional  instruction  without 
comparing  their  instructional  effectiveness.  Doughty  and  Lent  (1984)  present  one  such 
comparison  involving  the  actual  costs  of  the  two  approaches.  This  comparison  concerns  a 
training  facility  for  a  large  jet  engine  rebuilding  facility.  Doughty  and  Lent  estimated  the 
total  costs  for  procuring,  installing,  and  maintaining  interactive  videodisc  instruction  for 
this  facility  over  a  10  year  period  to  be  $4,419,000.  Three  sources  of  major  cost 
avoidances  to  be  realized  from  installation  of  this  interactive  videodisc  instruction  were 
identified  as  engine  failure  avoidance,  training  cost  avoidance,  and  staff  savings.  Doughty 
and  Lent  estimated  the  total  10  year  savings  from  these  three  sources  alone  to  be 
$14,031,000,  yielding  a  return  on  investment  of  316  percent. 

Many  commentators  have  remarked  on  the  advantages  of  computer-based 
instruction  and  interactive  videodisc  instruction  for  delivering  standardized,  decentralized 
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training  to  students-rather  than  delivering  students  to  the  training.  Walker  (1985) 
presented  an  industrial  training  study  in  which  the  costs  of  delivering  interactive  videodisc 
instruction  to  remote  sites  is  compared  to  the  costs  of  centralized  training.  When  the  initial 
investment  costs  for  developing  and  installing  the  interactive  videodisc  training  are 
amortized  over  3  years  Walker  showed  that  the  costs  per  student  are  $1,568  for  the 
centralized  training  and  $553  for  the  interactive  videodisc  training--a  cost  ratio  of  0.35  for 
combined  initial  investment  and  operating  and  support  costs. 

After  an  extensive  study  on  the  feasibility,  costs,  and  effectiveness  of  various 
methods  for  training  smog-check  mechanics,  Maher  (1988)  concluded  that  because  of  the 
elimination  of  instructors  and  reduced  training  time,  videodisc  instruction  would  provide 
the  most  cost-effective  approach  of  five  that  were  considered  for  hands-on  mechanic 
training  and  verification  testing.  Maher  found  that  the  costs  for  videodisc  training  would  be 
$50.60  per  student  compared  with  baseline  costs  of  $102.78  per  student— a  cost  ratio  of 
0.49  for  combined  initial  investment  and  operating  and  support  costs. 

As  Orlansky  and  String  (1977)  suggested,  an  important  source  of  cost  avoidance  in 
many  training  settings  is  student  time.  If  the  30  percent  savings  they  reported  for  students 
to  reach  threshold  levels  of  performance  using  computer-based  instruction  also  obtains  for 
interactive  videodisc  instruction,  and  it  was  found  in  this  review  to  be  about  31  percent, 
then  savings  for  some  high  student  load  courses  will  reach  millions  of  dollars.  A 
potentially  greater  source  of  savings  suggested  by  Solomon  (1986)  would  be  reductions  in 
the  number  of  people  the  military  has  to  support  in  a  job  category  if  the  time  needed  to  train 
people  for  it  can  be  reduced.  This  is  an  intriguing  suggestion,  but  it  requires  analyses 
beyond  the  scope  of  this  review. 

D.  COST-EFFECTIVENESS 

Interactive  videodisc  instruction  was  found  to  be  less  costly  in  nearly  all  studies  that 
reported  costs.  It  was  also  found  to  be  more  effective  overall  than  conventional  instruction. 
This  combination  of  findings  suggests  strongly  that  interactive  videodisc  instruction  may 
prove  to  be  the  most  cost-effective  alternative  for  instruction  in  many  applications. 
However,  this  result  is  best  viewed  as  suggestive  rather  than  as  conclusive  because  none  of 
the  reviewed  studies  examined  both  costs  and  effectiveness  in  a  properly  controlled 
empirical  study  based  on  a  systematic  model  of  cost  inputs  and  effectiveness  outputs. 

This  is  a  significant  consideration.  There  are  many  different  costs  that  may  or  may 
not  be  associated  with  an  instructional  approach.  Costs  reported  for  any  approach  will  vary 
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widely  depending  on  the  assumptions  and  procedures  used  to  collect— or  ignore— these 
different  costs.  If  these  assumptions  and  procedures  are  not  made  explicit,  as  they  often 
are  not,  it  is  difficult  to  judge  their  relevance  to  the  reader's  concerns,  and  it  is  impossible 
to  aggregate  them  across  different  studies-as  we  would  like  to  do  in  a  review  such  as  this — 
and  build  up  a  composite  understanding  of  what  they  might  be  for  an  approach  such  as 
interactive  videodisc  instruction. 

Budgets  are  inadequate  for  cost  estimation  because  they  do  not  include  all  the 
components  needed  for  an  adequate  cost  accounting;  they  may  distort  true  costs  of  a 
component  to  accord  with  local  accounting  practices;  and  they  represent  planned,  not  actual 
expenses.  After-the-fact  analyses  of  costs  are  inadequate  for  roughly  the  same  reasons, 
and  it  is  especially  difficult  to  disaggregate  costs  for  one  purpose  (e.g.,  cost  accounting) 
after  they  have  been  assembled  for  another  (e.g.,  production  control  and  auditing). 

Costs  and  effectiveness  data  should  be  drawn  from  the  same  comparative  study  for 
two  reasons.  First  and  most  obviously,  we  need  to  establish  the  same,  proper  experimental 
controls  for  all  conditions  (i.e.,  the  experimental  and  control  treatments)  for  which  cost  ana 
effectiveness  data  will  be  drawn.  Second,  we  need  to  ensure  that  the  same  cost  models  are 
used  for  all  treatments  and  that  they  are  related  to  the  same  effectiveness  models— which 
may  be  nothing  more  than  end  of  instruction  knowledge  or  performance,  but  which  must 
be  assessed  using  measures  common  to  all  treatments.  These  issues  are  discussed  more 
fully  by  Levin  (1983)  and  Fletcher  (1990). 

Nonetheless,  it  is  a  clear  finding  of  this  review  that  interactive  videodisc  instruction 
was  observed  to  be  both  less  costly  and  more  effective  across  the  studies  found  and 
examined.  Although  the  cost-effectiveness  of  interactive  videodisc  instruction  can  vary 
widely  depending  on  many  factors,  it  appears  that  there  will  be  many  instances  in  which 
interactive  videodisc  instruction  is  the  most  cost-effective  alternative. 

E.  TIME  ON  TASK 

Very  little  direct  evidence  exists  to  indicate  whether  students  using  interactive 
videodisc  instruction  are  willing  or  likely  to  spend  more  time  in  instruction  or  whether  the 
time  they  do  spend  is  more  task  centered  than  it  is  in  conventional  instruction.  A  number  of 
studies  listed  in  Appendix  A  surveyed  students'  opinions  of  the  interactive  videodisc 
instruction  they  received.  By  and  large  these  studies  report  that  the  students  enjoyed  the 
interactive  videodisc  instruction,  they  would  recommend  it  to  others,  and  they  found  it  easy 
to  use,  all  of  which  suggests  that  they  might  spend  more  time  on  relevant  instructional  tasks 
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when  they  use  interactive  videodisc  instruction.  Therefore,  it  is  not  unreasonable  to  expect 
interactive  videodisc  instruction  to  increase  time  on  task. 

Winkler  and  Polich  (in  press)  investigated  interactive  videodisc  instruction  used  to 
supplement  actual  equipment  experience  for  military  radio  operators.  They  found  that 
introduction  of  the  interactive  videodisc  materials  led  to  a  45  percent  increase  in  the  time 
students  spent  practicing  radio  installation.  These  students  received  increased  practice 
without  increasing  the  amount  of  total  time  they  spent  in  training.  This  result  is  promising, 
but  a  single  study  cannot  be  viewed  as  conclusive. 

F.  RETENTION 

Given  the  compelling  nature  of  much  interactive  videodisc  instruction,  it  seems 
reasonable  to  expect  it  to  be  memorable-skills  and  knowledge  obtained  under  interactive 
videodisc  instruction  should  be  retained  at  least  as  long  and  perhaps  longer  than  skills  and 
knowledge  obtained  under  other  instructional  approaches.  Four  of  the  studies  in  Appendix 
A  [Young  and  Tosti  (1981),  Bunderson  et  al.  (1984),  Verano  (1987),  and  Jones  (1988)] 
addressed  retention.  All  four  were  concerned  with  the  retention  of  knowledge,  and  Jones 
also  considered  the  retention  of  performance  skills.  Effect  size  (calculated  from  pooled 
standard  deviations)  for  the  single  measure  reported  by  Young  and  Tosti  increased  from 
-0.39  to  +0.14  over  a  retention  interval  of  10  weeks.  Effect  sizes  (calculated  from  pooled 
standard  deviations)  for  the  two  measures  reported  by  Bunderson  et  al.  reduced  from  an 
average  of  about  0.59  to  about  0.38  over  the  1-week  retention  interval  studied.  Effect  sizes 
(again  calculated  from  pooled  standard  deviations)  based  on  results  reported  by  Verano  for 
interactive  videodisc  instruction  over  "placebo",  non-instruction  reduced  from  2.94  to  1.33 
over  the  4-week  retention  interval  studied.  In  these  three  studies,  the  experimental  groups 
and  the  control  groups  seem  to  have  grown  together  over  the  retention  interval.  Jones 
reported  retention  data  but  not  end  of  instruction  achievement.  Her  data  indicate  substantial 
effect  sizes  remaining  in  favor  of  the  interactive  videodisc  training  2-3  weeks  after  the 
instruction  was  completed  for  both  knowledge  and  performance  outcomes. 

Some  research  suggests  that  retention  is  sensitive  to  the  choice  and  sequence  cf 
instructional  media.  Baggett  (1983)  showed  that  performance  on  assembly  tasks  assessed 
after  a  1-week  retention  interval  was  significantly  improved  for  students  who  had  hands-on 
practice  first  and  viewed  a  film  second  rather  than  the  other  way  around.  More  recent 
experimental  work  by  Baggett  (1988)  has  shown  no  advantage  in  retention  obtained  from 
hands-on  practice  presented  at  the  same  time  as  audiovisual  instruction-groups  without 
practice  performed  as  well  as  groups  with  practice  under  these  conditions.  She  did  show 
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an  advantage  in  both  immediate  and  1-week  delayed  performance  for  practice  and 
audiovisual  instruction  that  was  presented  sequentially  and  not  simultaneously.  The  best 
results  were  again  shown  in  groups  who  received  the  hands-on  practice  first  followed  by 
the  audiovisual  instruction  and  not  the  other  way  around.  This  finding  seems  in  keeping 
with  the  informal  observations  of  experienced  military  trainers  that  interactive  videodisc 
instruction  achieves  better  results  in  advanced  training  than  in  beginning  training. 


IV.  SUMMARY 


The  results  of  this  review  and  its  conclusions  principally  concern  the  instructional 
capabilities,  or  functionalities,  made  available  by  interactive  videodisc  systems.  Systems 
other  than  those  that  link  videodisc  players  to  computers  but  that  provide  the  same 
instructional  functionalities  may  achieve  at  least  the  level  of  effectiveness  observed  here.  At 
present  there  is  no  hardware  technology  that  can  competitively  provide  the  functionalities 
that  interactive  videodisc  systems  can,  but  digital  video,  digital  audio,  and  compact  disc 
technology  are  all  developing  rapidly  and  will  soon  claim  at  least  some  of  the  territory  now 
held  by  interactive  videodisc  systems,  The  findings  that  follow  are  based  on  capabilities, 
not  hardware,  and  should  obtain  for  any  system  that  provides  the  functionalities  now  found 
in  interactive  videodisc  instruction. 

A.  EFFECTIVENESS 

(1)  Interactive  videodisc  instruction  was  successfully  used  to  teach.  The  four 
comparisons  listed  in  Appendix  A  of  interactive  videodisc  instruction  with  a  "placebo" 
treatment  in  which  no  relevant  instruction  was  given,  resulted  in  an  average  effect  size 
calculated  using  pooled  standard  deviations  of  1.39  (an  increase  of  50th  percentile  students 
to  about  92nd  percentile  achievement)  and  an  average  effect  size  calculated  using  control 
group  standard  deviations  of  1.55  (an  increase  to  about  95th  percentile  achievement). 

(2)  Interactive  videodisc  instruction  was  more  effective  than  conventional 
approaches  to  instruction.  Forty-seven  comparisons  of  interactive  videodisc  instruction 
with  conventional  approaches  were  identified  for  this  review.  As  shown  in  Table  1  above, 
the  average  effect  size  for  these  47  comparisons,  calculated  using  pooled  standard 
deviations  over  all  instructional  settings,  instructional  approaches,  and  outcomes,  was 
found  to  be  0.50  (an  increase  for  50th  percentile  students  to  about  the  69th  percentile  of 
achievement). 

(3)  Interactive  videodisc  instruction  was  more  effective  than  conventional 
instruction  in  military  training.  Twenty-four  comparisons  of  interactive  videodisc 
instruction  with  conventional  approaches  to  military  training  for  subject  areas  that  included 
maintenance,  operator,  and  command  training  were  identified  for  this  review.  As  shown  in 
Table  2  above,  the  average  effect  size  for  these  24  comparisons,  calculated  using  pooled 


standard  deviations  over  all  instructional  approaches  and  outcomes,  was  found  to  be  0.38 
(an  increase  for  50th  percentile  students  to  about  the  65th  percentile  of  achievement). 

(4'  Interactive  videodisc  instruction  was  more  effective  than  conventional 
instruction  in  higher  education.  Fourteen  comparisons  of  interactive  videodisc  instruction 
with  conventional  approaches  to  instruction  in  higher  education  settings  were  identified  for 
this  review.  Over  all  instructional  approaches  and  outcomes,  there  was  an  average  inctease 
in  achievement  calculated  using  pooled  standard  deviations  of  0.69  standard  deviations  (an 
increase  for  50th  percentile  students  to  about  the  75th  percentile  of  achievement). 

The  lower  average  effect  sizes  found  for  military  training  than  for  higher  education 
may  be  due  to  a  focus  in  military  training  on  reaching  threshold  level(s)  of  achievement 
with  minimum  cost  and  time  so  that  students  who  reach  achievement  criteria  are  sent  on  to 
duty  assignments  rather  than  held  in  the  instructional  setting  as  they  are  in  higher  education. 
This  practice  contrasts  with  the  focus  in  education  on  maximizing  achievement  while 
holding  costs  constant  so  that  all  students  remain  in  instruction  for  roughly  the  same 
amount  of  time  allowing  more  efficient  instructional  approaches  to  yield  higher  levels  of 
achievement.  This  possibility  is  supported  by  the  eiiect  sizes  found  for  industrial  training, 
which  may  be  managed  in  ways  more  closely  resembling  military  training  than  higher 
education.  The  industrial  training  results  shown  in  Table  2  vary  widely.  However,  when 
the  effect  sizes  are  limited  to  knowledge  and  performance  outcomes  alone,  as  they  are  in 
Table  3  above,  the  results  from  military  and  industrial  training  are  much  closer  together  and 
both  differ  markedly  from  higher  education. 

(5)  Interactive  videodisc  instruction  was  equally  effective  for  both  knowledge  and 
performance  outcomes.  The  average  effect  sizes  calculated  using  pooled  standard 
deviations  for  27  knowledge  outcomes  (facts,  concepts,  and  other  information  students 
acquired)  and  for  20  performance  outcomes  (procedures,  skills,  and  other  capabilities 
students  could  demonstrate)  both  averaged  around  0.35  standard  deviations  (suggesting  an 
improvement  of  50th  percentile  students  to  about  the  64th  percentile). 

(6)  The  more  the  interactive  features  of  interactive  videodisc  technology  were 
used,  the  more  effective  the  resulting  instruction.  This  issue  was  examined  directly  by 
Verano  (1987)  who  compared  interactive  videodisc  instruction  with  segmented  videodisc 
instruction  and  linear  videodisc  instruction  and  found  consistently  greater  achievement  wtith 
greater  levels  of  interactivity.  The  issue  can  also  be  examined  by  comparing  the  overall 
effect  sizes  for  Level  II  videodisc  instruction  with  those  found  for  Level  III  videodisc 
instruction.  Table  6  (above)  shows  an  average  effect  size  of  -0.70  calculated  using  pooled 
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standard  deviations  for  13  results  from  Level  II  videodisc  instruction,  which  contrasts  with 
the  overall  effect  size  of  0.50  found  for  47  results  for  Level  III  videodisc  instruction  (Table 
1).  Effects  of  interactivity  can  also  be  examined  by  a  comparison  of  the  results  from 
Wankel's  (1984)  study  (Appendix  B)  with  those  from  Stevens's  (1984)  study  (Appendix 
A).  Both  studies  used  the  same  videodisc  materials,  and  both  tested  for  physics  knowledge 
outcomes.  Wankel's  study  used  Level  II  videodisc  materials  and  produced  an  effect  size  of 
-0.17,  Stevens's  study  used  Level  III  videodisc  materials  and  produced  an  effect  size  of 
0.29.  The  only  exception  to  this  pattern  of  greater  achievement  with  greater  interactivity 
occurred  in  one  of  Fowler's  (1980)  two  comparisons  of  branched  interactive  videodisc 
instruction  with  linear  videodisc  instruction  (Appendix  A).  This  comparison  resulted  in  an 
effect  size  of  -0.28  and  favored  linear  videodisc  instruction  --  the  other  resulted  in  an  effect 
size  of  0.72  and  favored  branched  videodisc  instruction.  Aside  from  this  single  result,  the 
pattern  of  greater  achievement  with  greater  interactivity  was  consistent  across  the  studies 
identified  for  this  review. 

(7)  Directed,  tutorial  approaches  were  more  effective  than  stand-alone  simulations 
in  interactive  videodisc  instruction.  This  issue  is  addressed  by  the  effect  sizes  shown  in 
Table  4  (above).  Similar  results  were  found  for  knowledge  outcomes,  performance 
outcomes,  and  knowledge  and  performance  outcomes  combined.  The  effect  sizes  averaged 
around  0.70  for  tutorial  approaches,  0.40  for  combined  tutorial  and  simulation  approaches, 
and  0.15  for  simulations  by  themselves.  Tutorial  guidance  appears  to  pay  off.  This  result 
is  consistent  with  a  body  of  literature  that  suggests  that  simulators  and  training  devices  are 
more  effective  when  incorporated  into  a  complete  training  system  than  when  they  are  used 
as  stand-alone  resources.  On  the  other  hand,  the  cost  arguments  for  simulation  over  actual 
equipment  use  are  sufficiently  cogent  that  it  may  still  be  the  most  favorable  alternative  from 
a  cost-effectiveness  standpoint. 

(8)  Within-group  variability  of  achievement  was  smaller  in  interactive  videodisc 
instruction  than  in  conventional  instruction.  In  nearly  every  comparison  of  Level  III 
interactive  videodisc  instruction  with  conventional  instruction,  the  effect  size  calculated 
using  pooled  standard  deviations  was  larger  than  the  effect  size  calculated  using  control 
group  standard  deviations.  This  result  suggests  that  achievement  in  the  interactive 
videodisc  groups  was  less  variable  and  more  equitably  spread  throughout  the  student 
population  than  it  was  in  the  conventional  instruction  groups.  It  has  been  suggested  that 
one  benefit  of  interactive  over  conventional  instruction  is  that  fewer  students  are  left  behind 
-  interactivity  insures  that  every  student  receives  some  individualized  attention  and  is 
therefore  either  goaded  or  aided  into  some  progress  toward  the  instructional  goals.  This 
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suggestion  is  supported  by  the  findings  shown  in  Tables  1-4.  It  is  also  anticipated  by 
commentators  on  individualized  instruction  and  aptitude-treatment  interaction  (e.g.,  Como 
and  Snow,  1986).  If  it  explains  the  pattern  of  standard  deviations,  it  will  be  particularly 
important  for  training  settings  in  which  the  intention  is  to  bring  every  student  to  criterion 
thresholds  of  performance  in  contrast  to  education  settings  in  which  maximizing  each 
student's  performance  is  more  likely  to  be  emphasized. 

(9)  The  effectiveness  of  interactive  videodisc  instruction  was  greater  than 
computer-based  instruction  has  been  found  to  be.  Both  approaches  have  been  found  to  be 
more  effective  than  conventional  approaches  to  instruction,  but  on  the  basis  of  studies 
reviewed  here,  interactive  videodisc  instruction  was  the  more  effective  of  the  two 
approaches.  The  average  effect  size  of  around  0.70  observed  for  interactive  videodisc 
instruction  used  in  colleges  is  considerably  higher  than  both  the  average  effect  size  of  0.26 
found  by  Kulik  and  Kulik  (1986)  for  computer  based  instruction  in  colleges  and  the 
average  effect  size  of  0.42  found  by  Kulik,  Kulik,  and  Shwalb  (1986)  for  computer-based 
instruction  in  adult  education. 

Videodisc  capabilities  increase  the  hardware  costs  of  a  standard  computer-based 
instruction  system  by  about  $2,500  and  the  software  or  instructional  preparation  costs  by 
widely  varying  amounts  depending  on  the  type  and  use  of  visual  information  in  the 
courseware.  One  might  well  argue  for  commensurate  increases  in  the  effectiveness  of  the 
instruction  being  delivered.  However,  these  are  initial  investment  costs  only.  The  life 
cycle  costs  of  computer-based  instruction  compared  to  videodisc  instruction  may  differ 
trivially,  if  at  all.  Nonetheless,  differences  in  costs,  instructional  objectives,  and 
instructional  approaches  across  all  possible  applications  make  blanket  recommendations  for 
one  or  the  other  of  these  technologies  ill-advised.  Recommendations  concerning  the  use  of 
computer-based  instruction  and  interactive  videodisc  instruction  in  specific  applications  are 
probably  best  made  on  a  case-by-casc  basis. 

(10)  It  appears  reasonable  to  conclude  on  the  basis  of  these  findings  that 
interactive  videodisc  instruction  is  a  promising  approach  for  both  training  and  education, 
but  there  was  little  in  the  reviewed  studies  to  indicate  how  interactive  videodisc  instruction 
achieves  its  success.  Generally,  interactivity  seemed  valuable  and  more  interactivity 
appeared  to  produce  more  achievement,  but  the  nature  of  this  relationship  (i.e.,  how  much 
of  what  sorts  of  interactivity  produce  what  kinds  of  achievement  in  what  amounts)  remains 
unknown.  Generally,  more  needs  to  be  learned  about  what  and  how  the  different 
instructional  features  of  interactive  videodisc  instruction  contribute  to  the  increased  student 
achievement  observed  across  the  studies  reviewed  here. 
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Further,  there  are  many  outcomes  such  as  speed  of  response,  accuracy  of  response, 
attitude  toward  the  subject  matter,  insight,  transfer,  and  retention  that  either  alone  or  in 
some  combination  may  become  the  objectives  of  instruction.  How  different  design 
alternatives  for  interactive  videodisc  instructio  'ontribute  quantitatively  to  these  outcomes 
(i.e.,  how  to  design  instruction  to  achieve  specific  outcomes)  remains  poorly  understood  at 
best.  Follow-on  research  might  begin  with  detailed  examination  of  the  interactive  videodisc 
instruction  materials  identified  by  this  review. 

B.  COST 

Interactive  videodisc  instruction  was  found  to  be  less  costly  than  conventional 
instruction  in  nearly  all  the  studies  reviewed  here  that  examined  costs.  All  13  cost  ratios 
(the  ratio  of  costs  for  interactive  videodisc  instruction  over  conventional  instruction) 
reported  in  Appendix  A  are  less  than  1.0  indicating  lower  costs  in  every  measured  instance 
for  interactive  videodisc  instruction.  The  average  across  these  13  cost  ratios  is  about  0.36. 
Most  of  the  low  cost  ratios  reported  in  Appendix  A  are  due  to  the  substitution  of  interactive 
videodisc  hardware  and  simulation  software  for  actual  equipment  hardware  in  training. 
These  are  favorable  results  for  the  use  of  interactive  videodisc  instruction,  but  it  should  be 
noted  that  they  were  all  calculated  from  different  categories  of  costs  and  from  a  variety  of 
cost  models.  It  seems  reasonable  to  conclude  that  interactive  videodisc  instruction  is  less 
costly  than  conventional  instruction  in  many  areas,  but  that  specific  costs  savings,  if  any, 
must  be  determined  on  a  case-by-case  basis. 

Another  area  of  cost  avoidances  involves  reductions  in  time  to  reach  criterion  levels 
of  achievement  in  instruction.  If  students  receive  pay  and  allowances  while  they  are  in 
school,  then  student  time  reductions  are  an  especially  important  source  of  cost  avoidances. 
The  average  time  saved  across  the  studies  covered  by  this  review  was  31  percent,  which 
matches  closely  the  30  percent  time  savings  reported  by  Orlansky  and  String  (1977)  for 
computer-based  instruction.  With  over  250,000  students  involved  in  formal,  residential 
training  each  day,  these  findings  promise  significant  reductions  in  military  training  costs. 
The  reductions  in  total  military  manpower  that  can  result  from  such  time  savings  are  likely 
to  be  an  additional  and  significant  source  of  cost  savings. 

C.  COST-EFFECTIVENESS 

Interactive  videodisc  instruction  was  found  to  be  both  less  costly  in  all  studies  that 
reported  costs  and  more  effective  overall  than  the  conventional  approaches  with  which  it 
was  compared.  This  finding  should  be  viewed  as  suggestive  rather  than  conclusive  since 
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no  studies  were  found  that  used  systematic  models  to  provide  empirical  data  on  both  cost 
inputs  and  effectiveness  outputs. 

D.  TIME  ON  TASK 

Interactive  videodisc  instruction  may  increase  time  on  task.  Intuitively,  the  low  cost 
of  interactive  videodisc  instruction  relative  to  the  costs  of  actual  equipment  should  mean 
that  instead  of  passively  watching  demonstrations  on  actual  equipment,  students  can  be 
actively  engaged  with  equipment  simulated  by  interactive  videodisc  technology-learning  by 
doing,  not  learning  by  watching.  Only  one  study  reported  an  observation  relative  to  this 
point.  Winkler  and  Polich  (in  press)  reported  a  45  percent  increase  in  the  time  spent 
practicing  radio  installation  as  a  result  of  the  introduction  of  interactive  videodisc.  This 
result  seems  best  viewed  as  promising,  but  not  conclusive. 

D.  RETENTION 

Based  on  evidence  presented  here,  there  is  little  reason  to  believe  that  interactive 
videodisc  instruction  will  have  a  significant  effect  on  retention  of  either  knowledge  or 
performance.  The  four  studies  in  Appendix  A  that  reported  retention  data  show  a  drift  of 
both  positive  and  negative  results  back  to  zero  (i.e.,  to  no  differences  in  achievement 
between  interactive  videodisc  instruction  and  conventional  instruction).  Post-training 
experiences  typically  influence  retention  of  knowledge  and  performance  in  a  powerful 
manner,  and  they  may  well  wash  out  any  effects  of  the  instructional  approach  used  to  teach 
the  knowledge  or  performance  in  the  first  place.  The  findings  for  interactive  videodisc 
instruction  in  this  review  corroborate  other  Defense  experiences  with  new  training 
approaches.  It  has  generally  been  found  that  their  promise  and  payoff  reside  primarily  in 
improving  the  efficiency  of  instruction,  not  in  assuring  that  what  is  learned  will  be  retained 
longer. 
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V.  CONCLUSION 


The  47  studies  reviewed  here  indicate  that  for  many  different  instructional  settings 
and  many  different  instructional  objectives,  interactive  videodisc  instruction  is  both  more 
effective  and  less  costly  than  conventional  instruction.  They  suggest,  therefore,  that 
interactive  videodisc  instruction  can  have  a  significant  positive  impact  on  the  productivity  of 
resources  allocated  to  military  training  and  education  and  on  the  availability  of  people 
needed  to  operate  and  maintain  military  systems.  Although  more  needs  to  be  learned  about 
how  interactive  videodisc  instruction  should  be  designed  and  employed,  it  should  now  be 
routinely  considered  and  used  in  military  training  and  education. 
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Table  A- 1 .  Studies  of  Level  Ill  Videodisc  Instruction. 
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T  --  Tutorial  Approach;  C  -  Tutorial  and  Simulation  Approach  Combined;  S  --  Simulation  Approach;  K  —  Knowledge  Measure; 
P  --  Performance  Measure;  R  -  Retention;  D  -  Time  to  Complete  Instruction;  KX  -  Knowledge  Average;  PX  -  Peformance 
Average;  RX  -  Retention  Average;  AX  -  Overall  Average  (for  K,  P,  R,  &  D  Measures);  I  --  Interactivity  Comparison;  IX  -- 
Interactivity  Average;  B  —  Baseline  Comparison;  BX  --  Baseline  Average 
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Notes  Concerning  Effect  Size:  ,  ^ 

T  -  Tutorial  Approach;  C  --  Tutorial  and  Simulation  Approach  Combined;  S  --  Simulation  Approach;  K  --  Knowledge  Measure; 
P  -  Performance  Measure;  R  --  Retention;  D  -  Time  to  Complete  Instruction;  KX  -  Knowledge  Average;  PX  -  Peformance 
Average;  RX  -  Retention  Average;  AX  -  Overall  Average  (for  K,  P,  R,  &  D  Measures);  I  --  Interactivity  Comparison;  IX  - 
Interactivity  Average;  B  -  Baseline  Comparison;  BX  -  Baseline  Average 
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Tabic  A- 1  (Continued).  Studies  of  Level  HI  Videodisc  Instruction. 
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Notes  Concerning  Effect  Size: 

T  -  Tutorial  Approach;  C  --  Tutorial  and  Simulation  Approach  Combined;  S  --  Simulation  Approach;  K  --  Knowledge  Measure; 
P  -  Performance  Measure;  R  -  Retention;  D  -  Time  to  Complete  Instruction;  KX  -  Knowledge  Average;  PX  -  Peformunce 
Average;  RX  -  Retention  Average;  AX  -  Overall  Average  (for  K,  P,  R,  &  D  Measures);  I  -  Interactivity  Comparison;  IX  - 
Interactivity  Average;  B  -  Baseline  Comparison;  BX  -  Baseline  Average 
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Notes  Concerning  Effect  Size: 

T  -  Tutorial  Approach;  C  --  Tutorial  and  Simulation  Approach  Combined;  S  -  Simulation  Approach;  K  -  Knowledge  Measure; 
P  --  Performance  Measure;  R  --  Retention;  D  -  Time  to  Complete  Instruction;  KX  --  Knowledge  Average;  PX  --  Peformance 
Average;  RX  -  Retention  Average;  AX  -  Overall  Average  (for  K,  P,  R,  &  D  Measures);  I  -  Interactivity  Comparison;  IX  - 
Interactivity  Average;  B  -  Baseline  Comparison;  BX  --  Baseline  Average  -  - 


Table  A- 1  (Continued).  Studies  of  Level  lit  Videodisc  Instruction. 
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Table  A- 1  (Continued).  Studies  of  Level  III  Videodisc  Instruction. 
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Table  B-l.  Studies  of  Level  II  Videodisc  Instructior,. 
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Notes  Concerning  Effect  Size: 

K  --  Knowledge  Measure;  P  —  Performance  Measure;  D  —  Time  to  Complete  Instruction 
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Notes  Concerning  Effect  Size: 

K  -  Knowledge  Measure;  P  -  Performance  Measure;  D  -  Time  to  Complete  instruction 


